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Training  and  Simulation 


FOREWORD 


Limited  time  and  wide  geographical  dispersion  of  both  units  and  individuals 
in  the  National  Guard  and  Army  Reserve,  i.e.,  the  Reserve  Component  (RC) ,  make  it 
difficult  and  costly  for  soldiers  to  travel  to  branch  schools  for  training. 
Therefore,  the  RC  is  exploring  alternatives  that  will  use  technology  to  bring 
training  and  educational  opportunities  to  the  soldiers'  homes.  One  of  these 
alternatives  is  the  creation  of  remotely  conducted  classes  in  which  individuals 
are  linked  with  each  other  and  their  instructors  using  asynchronous  computer 
conferencing. 

This  report  summarizes  the  findings  on  the  effectiveness  and  costs  of  using 
asynchronous  computer  conferencing  and  makes  recommendations  for  course  design 
and  teaching  in  the  RC.  The  report  was  developed  by  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI),  Boise  office  within  the 
charter  of  the  Training  Technology  Field  Activity,  Gowen  Field,  whose  mission  is 
to  improve  Reserve  Component  Training  effectiveness  and  efficiency  through  the 
testing  and  application  of  technology.  The  research  task  supporting  this  mis¬ 
sion,  "Application  of  Technology  to  Meet  Reserve  Component  Needs,"  is  organized 
under  the  "Training  for  Combat  Effectiveness"  program  area.  The  National  Guard 
Bureau,  Forces  Command,  and  Training  and  Doctrine  Command  Headquarters  (TRADOC 
HQ)  sponsored  this  project  under  the  Memorandum  of  Understanding  signed  12  June 
1985  that  established  the  office.  Project  results  have  been  briefed  to  TRADOC  HQ, 
Forces  Command,  Office  of  the  Chief,  Army  Reserve,  and  the  National  Guard  Bureau. 


)GA^K.  J0HNSC 
Technical  Director 


v 


DISTRIBUTED  TRAINING  FOR  THE  RESERVE  COMPONENT:  REMOTE  DELIVERY  USING 
ASYNCHRONOUS  COMPUTER  CONFERENCING 


EXECUTIVE  SUMMARY 


Research  Requirements: 

This  report  evaluates  the  cost-effectiveness  of  using  Asynchronous  Computer 
Conferencing  (ACC)  techniques  to  provide  high  quality,  remotely  delivered  train¬ 
ing  to  the  U.S.  Army  Reserve  Component  (RC)  and  develops  guidelines  for  effec¬ 
tively  conducting  such  training. 


Procedure: 

The  evaluation  used  a  portion  of  the  Engineer  Officer  Advanced  Course  (EOAC) 
as  a  test  bed.  Course  materials  that  taught  the  same  content  presented  in  the 
resident  course  were  developed  for  remote,  asynchronous  presentation.  This  de¬ 
livery  system  was  called  the  System  for  Managing  Asynchronous  Remote  Training 
(SMART) .  The  efficacy  of  remote  presentation  was  compared  to  that  of  the  resident 
program  with  regard  to  throughput,  performance,  acceptability,  and  cost. 

The  current  state  of  RC  training  and  the  potential  for  remote  training  were 
described,  as  was  previous  research  on  remote  learning.  A  summative  evaluation 
was  used  to  determine  cost-effectiveness  compared  to  resident  training.  A  forma¬ 
tive  evaluation  and  literature  review  were  used  to  develop  guidelines  for  con¬ 
ducting  ACC  training. 


Findings: 

The  summative  findings  were  as  follows: 

1.  ACC  training  was  more  cost-effective  than  resident  training  in  that 

a.  ACC  performance  (tests,  homework,  practical  exercises)  did  not 
differ  from  resident  performance; 

b.  Costs  for  development  and  operation  of  an  ACC  course  were  less  than 
operating  a  comparable  resident  course. 
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2.  Comparisons  of  the  amount  students  perceived  they  learned  during  the 
course  showed  that  ACC  students  felt  they  learned  more  than  did  the 
resident  students. 

The  formative  findings  were  as  follows: 

1.  The  imposition  of  deadlines  was  the  single  most  effective  method  of 
ensuring  that  students  worked  through  the  course  at  a  desirable  pace. 

The  use  of  group  activities  also  served  as  a  pacing  aid. 

2.  The  availability  of  group  interaction  opportunities  positively  impacted 
students'  motivation  to  participate  in  the  course. 

3.  Students  can  be  expected  to  spend  8  hours  per  week  working  on  a  SMART 
course.  This  8  hours  includes  administrative  time  for  activities  such 
as  interacting  with  the  computer  and  organizing  information. 

4.  Students  preferred  immediate  feedback  to  delayed  feedback.  If  feedback 
must  be  delayed,  the  course  structure  should  permit  students  to  move  on 
to  another  assignment  while  awaiting  feedback,  wherever  practical . 

5.  The  availability  of  support  communications,  such  as  a  telephone  hot¬ 
line,  is  critical  to  the  success  of  a  SMART  course. 


Utilization  of  Findings: 

This  report  presents  evidence  supporting  the  cost-effectiveness  of  ACC  for 
providing  remotely  delivered  training  to  the  RC.  It  also  provides  guidelines  for 
how  such  courses  should  be  implemented  to  maximize  throughput,  performance,  and 
acceptance. 
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DISTRIBUTED  TRAINING  FOR  THE  RESERVE  COMPONENT:  REMOTE 
DELIVERY  USING  ASYNCHRONOUS  COMPUTER  CONFERENCING 


Introduction 

Statement  of  the  Problem 

In  its  efforts  to  maintain  overall  readiness,  the  U.S.  Army  is  faced 
with  some  unique  challenges  in  training  its  two  Reserve  Components  (RC),  the 
Army  Reserve  and  the  Army  National  Guard.  Training  is  a  key  aspect  of 
readiness  and,  as  described  below,  it  is  many  times  more  difficult  to 
adequately  train  the  RC  than  it  is  to  train  the  Active  Army.  The  importance 
of  a  well  trained  RC  cannot  be  overemphasized,  for  the  RC  makes  up  more  than  50%  of 
the  total  Army  strength. 

Background  on  Training  the  RC 

Training  is  the  responsibility  of  the  unit  commander,  who  may  have  many 
different  military  occupational  specialties  (MOS)  to  train.  Typically,  RC 
commanders  have  only  39  training  days  available  each  year  distributed  over 
12  weekends  and  a  two-week  annual  training  session.  In  comparison,  their 
Active  Army  counterparts  train  their  units  continuously  throughout  the 
training  year. 

RC  units  are  scattered  across  the  United  States  and  overseas  at  more 
than  4,000  armories  and  reserve  centers.  As  a  result,  there  are  large 
numbers  of  soldiers  with  low  geographical  density.  Low  density  MOS  often 
do  not  justify  local  courses,  so  travel  to  resident/branch  schools  is 
required.  Resident  schooling  is  commonly  viewed  as  the  best  the  Army  has  to 
offer,  but  training  must  accommodate  civilian  job  ana  family 
responsibilities.  Thus,  a  trip  to  the  branch  school  may  not  be  an 
acceptable  alternative  for  many  members  of  the  RC. 

Current  RC  Education  Options 

Usually,  members  of  the  RC  complete  course  requirements  through  a 
combination  of  resident  branch  schools,  such  as  the  Engineer  School,  Reserve 
Forces  ( RF )  schools,  and  the  Army  Correspondence  Course  Program  (ACCP). 

Resident  training  yields  not  only  a  high  throughput  (number  of  soldiers 
completing),  but  also  high  quality  instruction.  E i si ey  and  Viner  (1989) 
conducted  a  nationwide  survey  of  RC  opinions  on  training  and  reported  that 
soldiers  prefer  resident  classes  to  other  training  options  because  they  are 
perceived  as  being  of  higher  quality  (i.e.,  as  having  good  instructors, 
course  content,  and  facilities).  However,  resident  courses  are  not  very 
accessible  to  the  RC  soldier,  who  must  juggle  job  and  family  schedules  to 
attend.  Gaining  attendance  at  resident  school  is  more  difficult  than 
gaining  attendance  at  other  options  (Eisley  and  Viner,  1989).  Further,  the 
cost  to  the  Army,  in  terms  of  travel,  per  diem,  and  pay,  is  quite  high.  The 
cost  per  RC  student  for  seven  modules  of  the  Engineer  Officer  Advanced 
Course  (EOAC)  is  estimated  by  Training  and  Doctrine  Command  Headquarters  to  be 
$40. 5K. 
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RF  School  training,  on  the  other  hand,  sacrifices  some  quality  control 
for  greater  availability  and  lower  costs.  RF  school  data  indicate  a  per 
student  cost  of  53.3K  for  one  phase  of  EOAC.  RF  courses  tend  to  be  easier 
to  fit  into  civilian  commitments.  Eisley  and  Viner  (1989)  reported  that 
soldiers  favor  RF  schools  with  respect  to  ease  of  meeting  schedules,  class 
openings,  and  funding  to  support  attendance.  However,  the  quality  of  RF 
courses  is  not  always  consistent  across  locations,  since  maintaining  a  pool 
of  qualified  instructors  is  difficult  for  some  MOS  (Allen  Corporation, 

1986). 

Correspondence  courses,  although  readily  available  and  low  in  cost  (548 
per  student  for  one  phase  of  EOAC,  Allen  Corporation,  1986),  have  much  lower 
throughput  and  quality  of  training  than  the  other  two  alternatives.  In  many 
content  areas,  ACCP  materials  may  not  be  doctrinally  or  technically  correct, 
due  to  long  revision  cycles  (Allen  Corporation,  1986).  Soldier  acceptance 
of  correspondence  training  is  lower  than  acceptance  of  any  other  training 
option  (Eisley  and  Viner,  1989).  As  a  result,  the  dropout  rate  for 
correspondence  courses  is  quite  high,  often  exceeding  50%  (Allen 
Corporation,  1986). 


Purpose 

An  ideal  training  option  for  the  RC  would  be  one  that  minimizes  cost 
and  maximizes  quality,  throughput,  and  avai 1 abi 1 i ty.  High  quality  training 
that  could  be  delivered  to  the  soldier's  home  or  armory/reserve  center 
should  provide  good  throughput  and  acceptance  and  meet  the  ideal  if  it  were 
affordable.  A  potential  method  of  providing  such  training  is  remotely 
delivered,  computer-mediated  training.  The  practical  implications  and 
cost-effectiveness  of  providing  such  training  had  been,  however,  largely 
unexplored. 

The  purpose  of  this  project,  then,  was  to  investigate  an  innovative 
alternative  to  RC  training:  the  System  for  Managing  Asynchronous  Remote 
Training  (SMART).  SMART  is  a  distributed,  computer-mediated  training  system 
capable  of  delivering  training  to  soldiers'  homes  and/or  armories/reserve 
centers.  Of  interest  are  the  quality,  acceptability,  throughput,  and  cost 
of  this  system.  Specific  requirements  were  to  (1)  evaluate  the  cost- 
effectiveness  of  using  Asynchronous  Computer  Conferencing  (ACC)  techniques 
to  provide  high  quality,  remotely  delivered  training  to  the  RC;  and  (2) 
develop  guidelines  for  conducting  such  training. 

Overview  of  SMART 


What  is  SMART? 

SMART  is  a  computerized  distributed  training  system,  the  functions  of 
which  are  to  provide: 

--  communication, 

--  a  combination  of  delivery  media,  and 
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--  course  management 

and  the  characteristics  of  which  are  to  support: 


--  geographically  distributed  training, 

--  asynchronous  and  synchronous  training,  and 
--  computer  mediated  training  and  communications. 

As  is  illustrated  in  Figure  1,  the  SMART  concept  provides  a 
communication  system,  allowing  an  instructor  to  communicate  with  students, 
students  to  communicate  with  the  instructor,  and  students  to  communicate 
with  each  other,  all  in  a  distributed  fashion.  Each  student  and  instructor 
has  a  personal  computer  that  is  networked  through  a  host  mainframe  computer 
using  existing  telephone  lines.  Other  names  for  this  kind  of  communication 
include  asynchronous  computer  conferencing  (ACC)  and  computer-mediated 
communication  (CMC). 

The  SMART  concept  also  provides  a  combination  of  delivery  media.  All 
types  of  instruction,  including  paper-based  materials,  computer  assisted 
instruction  (CAI),  storyboards  (computerized  slide  presentation),  formal  and 
informal  discussions,  problem  solving  groups,  peer  tutoring,  expert  groups, 
practical  exercises,  simulations,  video-  and  audio-tapes,  interactive  video 
disks  (IVD),  and  hands-on  activities,  can  be  accessed  via  SMART. 

Finally,  the  SMART  concept  provides  a  course  management  system, 
allowing  the  instructor  to  control  and  administer  lessons,  exercises,  and 
tests,  while  maintaining  rosters,  grade  books,  and  attendance  records. 

SMART  also  provides  mechanisms  by  which  the  instructor  can  provide  feedback 
on  the  performance  of  various  activities  to  soldiers. 


The  distributed  nature  of  SMART  means  that  geographically  distant 
learners  can  participate  in  the  instruction  without  the  need  to  come 
together  in  one  location.  Students  and  instructional  staff  can  be  located 
anywhere  there  is  access  to  a  telephone.  Students  can  work  from  their 
homes,  armories,  or  reserve  centers,  and  instructors  can  teach  from  branch 
schools,  RF  schools,  or  their  homes.  With  portable  computers,  students  and 
instructors  can  continue  their  participation  in  classes  even  when 
travel  ling. 


Most  of  the  instruction  delivered  via  SMART  is  accessed 
asynchronously.  That  is,  not  everyone  must  participate  at  the  same  time. 
This  flexibility  of  scheduling  makes  SMART  quite  adaptive  to  personal  time 
constraints.  However,  it  also  means  that  there  are  built-in  time  delays 
before  all  students  receive  the  instruction  and  that  certain  activities, 
such  as  group  discussions,  will  take  longer  than  they  would  in  the 
face-to-face  environment.  When  time  delays  are  not  practical,  SMART  allows 
for  synchronous  communication.  Here,  all  students  access  SMART  at  one  time 
and  work  together  to  accomplish  a  given  task.  When  the  task  is  completed, 
they  return  to  the  asynchronous  mode. 
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Figure  1.  The  SMART  System 
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Because  SMART  is  computer-mediated,  it  requires  that  each  student  have 
a  computer  with  a  modem,  to  enable  computer  communication  over  the 
telephone,  and  appropriate  software  to  facilitate  computer  communication  and 
learning  activities.  It  also  requires  telephone  links,  a  host  computer 
which  supports  the  uploading  (transmitting  to  someone)  and  downloading 
(receiving  from  someone)  of  information,  and  a  certain  amount  of  learning  on 
the  part  of  the  user.  The  system  facilitates  meaningful,  connected 
discussions  by  a  group  of  students.  These  discussions  take  place  on  their 
computers  as  students  observe  written  input  from  each  other  and  comment  upon 
it.  This  distributed  training  system  also  allows  instructors  and  schools  to 
take  advantage  of  the  fact  that  every  student  has  a  computer,  facilitating 
the  use  of  computer-based  training. 

The  Distributed  Training  Situation:  Students  and  Instructors 

As  in  some  of  the  other  distributed  learning  situations,  such  as 
correspondence,  students  learning  via  SMART  are  experiencing  a  tabletop 
learning  environment  --  they  are  likely  to  be  sitting  at  the  kitchen  table 
or  at  a  desk  in  their  homes,  working  alone  in  the  sense  of  not  having  other 
students  physically  present.  Because  they  are  in  the  RC,  they  are  fitting 
their  training  around  their  daily  routines  and  different  students  may  be 
working  on  very  different  schedules.  These  factors  combine  to  make  the 
SMART  learning  experience  quite  different  from  that  of  a  student  taking  a 
resident  course.  Figure  2  depicts  segments  of  a  "day  in  the  life"  of  a 
resident  student  and  a  SMART  student,  highlighting  differences  between  the 
two.  Appendix  A  shows  what  the  SMART  student  sees  and  does  during  a  SMART 
computer  session. 

The  fact  that  SMART  is  characterized  by  being  distributed,  asynchronous 
and  synchronous,  computer-mediated  training  and  communication  also  impacts 
the  SMART  instructor  with  respect  to  both  work  schedule  and  duties 
performed.  The  emphasis  on  computer  mediation  and  independent  learning 
shifts  the  emphasis  of  the  role  of  the  instructor  from  primarily  that  of  a 
deliverer  of  instruction  to  more  of  that  of  a  facilitator  and  a  counsellor. 
That  is,  the  SMART  instructor  spends  less  time  providing  course  content  via 
lecturing  or  other  classroom  techniques  than  does  the  resident  instructor 
and  more  time  guiding  students  through  the  learning  experience  by  directing 
their  studies,  answering  questions,  and  providing  performance  feedback. 

Figure  3  gives  highlights  of  a  "week  in  the  life"  of  a  SMART 
instructor.  For  the  purposes  of  the  illustration,  it  is  assumed  that  the 
instructor  works  full  time  teaching  one  course.  Note  that,  although  the 
instructor  works  for  40  hours  during  the  week,  that  time  is  allocated 
according  to  student  needs  rather  than  as  an  eight  to  five  workday. 

Further,  to  ensure  timely  feedback,  it  is  necessary  for  the  instructor  to 
check  in  on  the  computer  on  at  least  a  daily  basis. 
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Typical  Day 

Resident  Student 

SMART  Student 

Notes 

0700 

Wake  up  &  take 
a  shower 

Drive  to  work 

The  resident  student 
can  devote  the  entire 
day  to  course  work; 

0800 

Report  to  the 

Arrive  at  the 

the  SMART  student 

first  class  of 

office  &  begin  the 

devotes  more  than  a 

the  day 

work  day 

third  of  the  day  to 
civil ian  employment 

0930 

Take  a  break  & 
chat  with  other 
students 

Try  to  reach  a 
classmate  by  phone; 
no  luck 

0945 

Begin  a  small 

Get  work  assignment 

The  resident  student 

group  exercise 

to  write  a  major 
report  by  Friday 

has  face-to-face 
interaction  with 
classmates  so  group 

1145 

Finish  the 

Classmate  returns 

exercises  are 

group  exercise; 

cal  1 ;  spends  20 

accomplished  quickly 

break  for  lunch 

minutes  scheduling 
a  synchronous 
meeting  for  Friday 

in  a  pre-scheduled 
time  period;  the 

SMART  student  must 
coordinate  group 

1300 

Complete  a  CAI 

Eat  lunch  at  desk 

work  adding  admin 

lesson  at  the 
computer  lab 

while  doing  a  SMART 
reading  assignment 

time  to  the  activity 

1400 

Go  to  lecture 

Resume  office  work 

1700 

Return  to  quarters 

Drive  back  home 

1800 

Have  dinner 
with  classmates 

Have  dinner  with 
family 

1930 

Do  a  homework 
assignment 

Check  kids' 
homework 

2040 

Watch  Monday 

Work  on  SMART--  do 

Only  after  meeting 

night  football 

a  CAI  and  a  paper- 
based  exercise 

civilian  job  & 
family  requirements 
can  the  SMART  student 

2330 

Go  to  bed 

Go  to  bed 

begin  his  or  her 
studies 

Figure  2.  A  day  in  the  lives  of  a  resident  and  a  SMART  student 
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SMART  Instructor 

Notes 

Monday 

0800-1100  Go  online  &  receive  messages  & 

Like  their  students 

scores  on  activities,  write 

instructors  access  SMART 

replies  &  feedback  &  upload 

via  the  task  list;  they 

(download/upload) 

can  see  grades  on  lessons 

1100-1500  Update  records  of  soldiers' 

that  are  computer  scored, 

progress;  phone  students  who 

pick  up  assignments  to 

have  fallen  behind 

grade,  post  grades, 

1500-1700  Take  hotline  call  from  a 

receive  questions  & 

student  having  software 

comments  &  post  answers 

problems;  send  new  disks 

to  them;  students  who  are 

not  getting  online  either 

Tuesday 

because  they  have  fallen 

0800-1000  Download/upload 

behind  or  are  having 

1000-1100  Obtain  clarification  from 

computer  problems  must  be 

proponent  school  on  topic 

contacted  by  phone 

about  which  soldiers  have 

questions 

In  SMART,  the  instructor 

has  the  opportunity  to 

consult  with  experts  or 

Wednesday 

written  resources  before 

0800-1100  Download/upload 

answering  a  question 

Thursday 

0800-1200  Download/upload 

The  time  needed  for 

1200-1600  Grade  exams  &  homeworks 

downloading  and  uploading 

will  vary  by  volume 

Friday 

0800-1100  Download/upload 

1200-1400  Add  funds  to  computer  accounts 

SMART  instructors  do  both 

1800  Go  online  for  synchronous 

class  admin  (grading. 

meeting 

etc.)  and  computer  admin 

1830-2200  All  students  are  ready  to  begin 

tasks 

the  exercise;  give  the 

assignment  &  watch  progress 

Synchronous  activities 

2200-2300  Grade  group  assignment  &  put 

must  be  scheduled  around 

feedback  on! ine 

students'  other  activities 

&  prompt  feedback  is 

Saturday 

critical  in  SMART 

0800-1100  Download/upload 

Sunday 

0800-1100  Download/upload 

To  ensure  timely  turn- 

1900-2100  Go  online  for  office  hours 

around  instructors  must 

be  on  the  computer  daily 

Figure  3.  A  week  in  the  life  of  a  SMART  instructor 
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Background 


The  literature  was  reviewed  to  determine  those  variables  most  likely  to 
maximize  throughput  and  quality  (performance  and  acceptability)  and  to 
minimize  cost  in  distributed  training.  Most  of  the  literature  describes 
public  civilian  education  findings  in  what  is  commonly  called  "distance 
education."  However,  much  of  this  literature  is,  nevertheless,  applicable 
to  military,  distributed,  asynchronous  computer-mediated  training.  The 
following  tables  summarize  the  findings  of  that  review.  (See  Wells,  199C, 
for  the  full  review.) 

In  addition,  a  pilot  test  was  conducted  to  determine  the  feasibility 
and  logistics  of  remotely  delivering  a  computer-mediated  course  for  the  RC, 
and  to  begin  to  identify  the  important  throughput,  quality,  and  cost 
issues.  A  brief  description  of  this  pilot  test  follows  the  literature 
summary. 

Literature  Review 


Distance  education  is  a  worldwide  phenomenon.  Remote  delivery  of 
education  is  popular  where  students  are  geographically  dispersed  or  rugged 
climate  conditions  prevail  (for  example,  in  Canada,  Norway,  and  Sweden). 
Third  world  countries  use  distance  education  in  an  attempt  to  educate  their 
people  in  the  most  economical  way  possible  and  developed  nations,  such  as 
the  United  States,  provide  distance  education  to  adults  who  cannot  easily 
attend  traditional  universities  because  of  geographic  dispersion  or  heavy 
obligations  to  career  and  family.  Thus,  the  principles  of  distance 
education  are  derived  from  a  wide  variety  of  applications,  both  academic  and 
industri al . 

While  this  summary  examines  variables  affecting  successful  remote 
delivery  of  education  in  general,  it  also  considers  the  effect  of  these 
variables  on  a  special  form  of  remote  delivery,  asynchronous  computer 
conferencing  (ACC).  ACC  is  the  computer  communications  medium  on  which 
SMART  courses  are  based  --  students  in  SMART  courses  communicate  with  one 
another  and  with  the  instructor  via  a  computer  conference.  Thus,  SMART  is 
just  a  specialized  form  of  distance  education.  Distance  education 
principles  should  apply  to  SMART  in  the  same  fashion  as  they  apply  to  other 
remotely  delivered  instruction. 

This  literature  summary  is  presented  in  tabular  form  and  is  divided 
into  three  major  sections.  Table  1  addresses  throughput,  which  concerns 
variables  affecting  dropout  and  rate  of  progress  through  a  course,  quality, 
the  variables  affecting  outcomes  such  as  level  of  performance  and  student 
acceptance,  is  discussed  in  Table  2.  Finally,  variables  affecting  cost  are 
shown  in  Table  3.  If  a  variable  affects  more  than  one  of  these  categories, 
it  is  listed  in  each  category. 
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Table  1 

Variables  Affecting  Throughout 


_ Finding _ 

Student-to-student 
interaction  decreases 
dropouts,  thus  enhancing 
completion  rates;  group 
interaction  also  facili¬ 
tates  student  study 
rates. 

Students  with  lower 
educational  levels  are 
more  likely  to  drop  out. 


Personal  contact  with  an 
instructor,  whether 
student-  or  instructor- 
initiated,  has  a  positive 
impact  on  throughput. 

Course  completion  is 
positively  related  to 
prompt  feedback  by  the 
instructor. 

The  rate  of  assignment 
completion  can  help 
predict  course  comple¬ 
tion;  completion  of 
assignments  in  the  first 
one-third  to  one-half 
of  the  course  is  espe¬ 
cially  critical  to  dropout 
reduction. 

Increasing  the  variety 
of  media  lowers  the  drop¬ 
out  rate. 


_ Imp! icati ons _ 

Group  interaction  oppor¬ 
tunities  should  be  incor¬ 
porated  in  course  design. 


Educational  level  can  be 
used  to  predict  not  only 
which  students  are  likely 
to  succeed,  but  also  which 
students  might  need  addi¬ 
tional  assistance  to 
succeed. 

Course  design  should 
incorporate  instructor- 
student  interaction. 


Course  materials  should 
be  designed  with  type 
and  frequency  of  feedback 
in  mind. 

To  facilitate  completion, 
increase  the  frequency 
and  decrease  the  length  of 
assignments. 


Use  a  full  range  of  media 
to  enhance  throughput. 


References 


Baath  (1982); 

Pentz  and  Neil  (1981) 


Wells  (1989); 
Woodley  and  Parlett 
(1983) 


Coldeway,  MacRury,  and 
Spencer  (1980); 

Woodley  and  Parlett 
(1983) 


Coldeway  (1980); 
Rekkedal  (1982) 


Wong  and  Wong  (1979) ; 
Woodley  and  Parlett 
(1983) 


Pentz  and  Neil  (1981) ; 
Woodley  and  Parlett 
(1983) 
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Table  1  (continued) 


_ Finding _ 

Student  dropout  is  higher 
under  conditions  of 
individual  rather  than 
instructor  pacing . 

Larger  class  size  can 
lead  to  less  feedback, 
slower  response  times, 
and  higher  student 
attrition. 


_ Imp! ications _ 

Externally  set  deadlines 
should  be  used  in  dis¬ 
tance  courses  to  facili¬ 
tate  completion. 

Twenty-five  students  is 
probably  a  reasonable 
class  size,  but  this  may 
vary  as  a  function  of 
media  selection  and 
course  content. 


References 


Coldeway  (1982); 
Morris,  Surber,  and 
Bijou  (1978) 


Hiltz  and  Turoff 
(1978);  Manock 
(1984);  Woodley  and 
Parlett  (1983) 
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Table  2 


Variables  Affecting  Quality  (Performance  and  Acceptability! 


_ Finding _ 

With  respect  to  student 
motivation:  students 
using  ACC  reported 
trying  to  do  a  more 
thorough  job  on  assign¬ 
ments  because  other 
students  would  have 
access  to  them;  motiva¬ 
tion  plays  a  role  in 
both  user  satisfaction 
and  in  acquisition  and 
retention  of  information. 

Ready  access  to  a 
computer  is  almost  a 
pre-requisite  for  suc¬ 
cessful  performance  in 
ACC;  qual ity  of  access 
(quiet  location,  etc.) 
also  impacts  performance. 

Learning  skills  and 
strategies  impact  per¬ 
formance;  cognitive 
processing  skills  (i .e. , 
logically  connecting 
ideas)  are  more  impor¬ 
tant  than  mechanical 
skills  (i.e.,  memoriza¬ 
tion);  students  do  not 
always  use  the  materials 
as  intended  by  the 
designer. 


_ Imp! ications _ 

Motivation  may  be  the 
most  important  variable 
in  utilization  of  a 
computer  conferencing 
system--even  a  wel 1  - 
designed  system  will  not 
be  used  without  positive 
motivation. 


Each  student  should  have 
a  computer  that  can  be 
set  up  in  a  convenient 
1 ocation . 


Instructional  materials 
should  be  designed  to 
encourage  depth  of 
processing;  course 
designers  should  use 
methods  to  ensure  that 
students  proceed  in  the 
proper  sequence. 


_ References 

Baath  (1982);  Hiltz 
(1986) ;  Kerr  and 
Hiltz  (1982);  Mason 
(1988);  Vallee, 
Johansen,  Lipinski, 
Spangler,  Wilson  and 
Hardy  (1975) 


Harasim  (1986) ; 
Kirkwood  (1988); 
Vallee,  Johansen, 
Randolph,  and  Hastings 
(1974) 


Marton  and  Svensson 
( 1982) ;  Rumble  and 
Keegan  (1982) 
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Table  2  (continued) 


Find  in 


ImDl ications 


References 


The  instructor  can  play  Instructors  must  adopt  a  Coldeway  (1980); 

a  critical  role  in  facilitative  role  in  ACC  Holmberg  (1981);  Kerr 

student  performance;  the  and  must  take  care  to  not  and  Hiltz  (1982); 

role  of  the  instructor  dominate  the  interaction;  Rekkedal  (1982);  Smith 

in  a  distance  environ-  specialized  instructor  (1988) 

ment  should  be  facili-  training  which  emphasizes 

tative  rather  than  the  differences  between 

authoritarian;  the  distance  and  face-to-face 

optimal  amount  of  training  and  which 

instructor-student  inter-  provides  skills  needed  in 

action  is  curvilinear:  the  distance  environment 

up  to  a  point,  user  is  needed. 

acceptance  is  facilitated 

by  high  instructor 

responsiveness,  but  too 

much  instructor  activity 

can  lead  to  student 

resentment . 

Group  interaction  may  Incorporate  group  acti-  Harasim  (1986) 

enhance  user  acceptance,  vities  to  enhance  student 

since  adults  prefer  to  acceptance. 

learn  from  respected 

peers  than  from  an 

authority  figure. 

Time  requirements  of  10  Students  should  be  re-  Kaye  (1981) 

to  15  hours  per  week  quired  to  spend  a  maximum 

should  be  acceptable  to  of  10  hours  per  week  on  a 

most  students,  accommo-  course;  course  duration 

dating  their  job  and  should  be  set  accordingly. 

family  commitments. 


Self-pacing  may  result  Hiltz  (1986) 

in  some  students  being 
weeks  behind  others,  with 
the  result  that  they  are 
not  prepared  to  perform 
the  most  recent  assign¬ 
ment. 
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Table  3 

Variables  Affecting  Cost 


Findinc 


Impl ications 


References 


Cost  per  student  increases  Personalized  feedback,  Rumble  (1988) 
if  more  personalized  although  costly,  enhances 

feedback  is  provided.  both  feedback  and  perfor¬ 

mance  . 


Development  costs  to  High  quality  materials  Rumble  (1988) 

produce  high  qua! itv  enhance  both  student 

material s  are  high,  and  motivation  and  perfor- 

may  exceed  those  for  mance;  materials  should  be 

resident  training;  use  produced  specifically  for 

of  existing  materials  the  distance  environment. 

reduces  costs,  but 

rarely  produces  good 

sel f- instructional 

material s . 


Media  selection  will  Use  of  a  variety  of  media  Rumble  (1988); 

impact  costs,  with  some  enhances  throughput,  so  Sparkes  (1984) 

media  being  cheaper  to  media  selection  should 

develop  than  others  not  be  driven  solely  by 

(i.e.,  a  one-hour  text  cost. 

assignment  will  cost 

less  than  a  one-hour 

interactive  video) . 


The  most  cost-efficient  Some  MOSs  may  involve  Rumble  (1988) 

distance  education  system  relatively  few  soldiers, 

is  one  with  few  courses  making  it  difficult  to 

and  large  class  sizes  achieve  an  economy  of 

and  in  which  classes  are  scale;  doctrinal  changes 

not  revised  frequently.  may  require  frequent 

updates. 
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In  summary,  the  research  reported  suggests  that  quality  instructional 
design  and  responsive  implementation  (including  instructor  performance)  may 
often  override  student  constraints  (like  time  pressure)  which  would 
otherwise  lead  to  dropout  (Woodley  and  Parlett,  1983).  Thus,  the  primary 
responsibility  for  student  retention  and  performance  may  rest  with  the 
quality  of  an  institution's  instructors,  courses,  and  support  services. 
Hence,  a  great  deal  of  attention  was  devoted  to  these  issues  in  the  current 
project. 

Pilot  Project  on  Using  SMART  for  Delivery  of  RC  Training 

Pilot  research  was  conducted  to  determine  whether  SMART  had  enough 
potential  for  RC  training  to  warrant  more  detailed  investigation.  Results 
of  this  project  are  reported  here,  since  they  served,  in  part,  as  the  basis 
for  the  decisions  made  in  the  implementation  test.  (See  Richards,  Hahn, 
Phelps,  Blackman  and  Schurman,  1989,  for  a  complete  report.) 

The  pilot  project  used  Asynchronous  Computer  Conferencing  (ACC)  to 
augment  Phase  III  of  the  U.  S.  Army  Reserve  Forces  School  version  of  the 
Engineer  Officer  Advanced  Course  (EOAC).  The  phase  consisted  of  seven 
engineer  subcourses  offered  through  the  Army  Correspondence  Course  Program 
(ACCP).  Topics  included  technical  subjects  such  as  soils  Engineering  and 
Military  Bridges.  Students  in  the  pilot  test  used  computers  to  communicate 
with  the  instructor  and  each  other  to  discuss  the  content  presented  in  the 
ACCP  materials  and  additional  requirements  developed  by  the  instructor.  The 
experimental  course  ran  from  November  1986  to  June  1987. 

The  course  management  function  of  SMART  was  limited  primarily  to 
providing  feedback  on  online  activities.  This  feedback,  however,  was 
certainly  more  rapid  and  frequent  than  that  afforded  by  ACCP.  Students  were 
required  to  interact  directly  with  the  computer  conference  host  computer; 
SMART  did  not  automatically  prepare  data  for  uploading  or  sort  downloaded 
data  for  them. 

Research  questions.  The  research  questions  addressed  in  the  pilot 
project  were  specific  and  limited: 

•  Can  the  throughput  and  quality  of  Army  correspondence  courses  be 
improved  by  the  addition  of  ACC? 

•  What  are  the  costs, logistical  requirements,  instructional  and/or 
motivational  methods  (such  as  computer  hardware/software  and 
instruction  methods)  necessary  for  implementation  of  ACC  for  RC 
training? 

'he  Throughput/Quality  issue  was  examined  by  comparing  two  versions  of 
Phase  III,  EOAC.  In  the  control  version,  participants  took  the  module  via 
correspondence  according  to  ACCP  guidelines.  In  the  experimental  version, 
the  ACCP  materials  were  also  presented,  but  they  were  augmented  by  ACC.  The 
dependent  measures  for  this  comparison  were  the  percent  of  students 
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completing  the  course,  the  speed  with  which  they  completed  (throughput),  and 
student  performance  (quality). 

To  examine  the  Cost/Logistics/Implementation  issues,  the  experimental 
version  of  the  course  was  analyzed  as  a  case  study.  The  activities  of  the 
students  and  the  instructor  were  documented  with  special  attention  to  cost 
and  logistic  factors  and  to  the  results  of  several  instructional  and 
motivational  manipulations  used  throughout  the  course. 

Lessons  learned  from  the  pilot  test.  While  many  of  the  results 
relating  to  throughput  and  quality  were  not  statistically  significant,  they 
did  suggest  that  further  investigation  of  computer  delivery  of  RC  training 
was  warranted:  more  students  completed  the  course  and  in  less  time  and 
there  was  a  promise  of  higher  level  of  knowledge  gained  and  retained  with 
ACC  than  with  ACCP  alone.  Specific  conclusions  were: 

(1)  The  experimental  group  showed  higher  rates  of  completion 
(percentage  of  subcourses  completed)  throughout  the  class  than  did 
the  control  group; 

(2)  The  experimental  group  also  had  a  higher  percentage  of  completion 
of  the  course  as  a  whole  than  did  the  control  group;  of  the 
original  experimental  group,  36%  completed  the  course  compared  to 
27%  of  the  control  group;  of  the  starters  in  each  group,  64%  of 
the  experimental  group  completed  vs  54%  for  the  control  group; 

(3)  The  experimental  group  students  who  completed  the  course  did  so 
earlier  than  the  control  group  completers;  there  was  a 
statistically  significant  difference  between  the  groups  on  number 
of  days  required  to  complete  the  course;  there  was  also  a 
difference  in  the  number  who  completed  the  course  on  time  (by  the 
planned  deadline)  for  each  group,  as  compared  to  those  who 
finished  by  the  final  deadline;  and 

(4)  The  on-time  finishers  in  the  experimental  group  did  better  than 
the  control  group  finishers  on  both  post-  and  retention-tests; 
late  finishers  in  the  experimental  group  performed  slightly  better 
than  the  control  group  on  the  post-test  and  did  better  on  the 
retention  test;  however,  performance  differences  were  not 
significant  since  ceiling  effects  were  evident  in  the  scores  for 
both  groups. 

As  shown  in  Table  4,  costs  for  the  computer-augmented  course  fall 
between  those  for  correspondence  and  resident  training.  If  this  result  is 
coupled  with  the  improved  throughput  and  quality  (over  correspondence) 
discussed  previously,  it  appears  that  the  question  of  ACC-mediated,  remotely 
delivered  training  warrants  further  study. 
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Table  4 


Cost  Comparison  for  Correspondence,  Correspondence  plus  ACC,  RF  School,  and 
Resident  School  Training  for  Phase  III  of  EOAC _ 


Training  Method 

Cost  per  Soldier 

Correspondence 

$48 

RF  School 

$3335 

Correspondence 

plus  ACC 

$4195 

Resident  School 

$11585 

Data  for  non-ACC  training  provided  by  TRADOC  Headquarters  or  the  1155th  RF 
School . 


The  logistics  and  implementation  issues  considered  dealt  primarily  with 
usability  and  acceptability  of  the  ACC  system.  Host  soldiers,  regardless  of 
previous  experience,  were  able  to  set  up  and  operate  their  computers.  In 
general,  soldiers  in  the  experimental  group  liked  using  the  computers.  On  a 
five-point  scale,  soldiers  rated  SMART  presentation  at  3.8,  below  the  rating 
of  4.0  for  resident  school,  but  above  the  rating  of  3.1  for  correspondence 
courses  alone.  ACC  was  rated  almost  the  same  as  RF  Schools  (3.8  vs  3.7, 
respectively) . 

Logistical  lessons  learned  included: 

(1)  Keep  both  hardware  and  software  as  simple  as  possible;  soldiers 
are  not  computer  experts  and  problems  in  these  areas  will  be  extremely 
frustrating  and  may  lead  to  dropouts; 

(2)  Support  communications,  such  as  a  telephone  hotline,  to  assist 
soldiers  who  are  having  difficulties  are  essential; 

(3)  Take  into  account  travel,  odd  work  schedules,  and  family  plans  of 
part-time  students  by  allowing  some  flexibility  in  scheduling; 

(4)  Individually  accomplished  activities  are  more  accommodating  to 
personal  schedules  than  are  group  activities; 

(5)  Time  requirements  must  be  reasonable  for  the  part-time  student; 

(6)  Conferencing  must  be  an  integral  part  of  the  course  requirements, 
not  a  sideline  to  a  course  primarily  conducted  in  some  other  fashion;  and 

(7)  Participation  should  be  integral  to  completion  of  course 
requirements;  soldiers  should  be  judged  on  the  basis  of  completion  of 
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learning  activities  (such  as  participating  in  a  group  exercise)  rather  than 
strictly  on  the  number  of  comments  entered. 

Rationale 


Based  on  the  encouraging  results  of  the  pilot  project,  a  more  rigorous 
implementation  test  of  SMART  was  conducted  to  study  throughput,  performance, 
acceptability,  and  cost  of  a  SMART  course.  The  general  approach  was  to 
develop  a  "best  shot"  course,  that  is  a  course  which  was  designed  to 
maximize  throughput,  performance,  and  acceptability.  The  literature  on 
remote  training  and  distance  education,  general  principles  of  instructional 
design,  and  experiences  from  the  pilot  project  were  used  to  determine  the 
specific  factors  investigated  within  the  course  and  to  guide  the  course 
design. 

Course  design  parameters  were  specified  at  the  outset  of  the  course, 
with  no  a  priori  intention  to  systematically  alter  them  as  the  course 
progressed.  However,  since  it  is  recognized  that  course  effecti veness  is 
largely  governed  by  good  course  design,  we  did  allow  formative  changes  in 
these  parameters  as  need  dictated.  These  formative  changes  were  documented 
as  part  of  the  "lessons  learned"  from  the  project. 

Several  independent  variables  (including  group  interaction,  pacing, 
instructor  characteristics,  and  type  of  course  requirements)  were 
systematically  manipulated  during  the  course.  Again,  formative  changes  were 
allowed  as  needed,  to  maximize  success.  (A  full  description  of  both  the 
course  design  parameters  and  the  independent  variables  is  contained  in  the 
"Method"  section.) 

Because  the  results  of  the  pilot  project  were  encouraging  and  because  a 
major  effort,  guided  by  established  literature,  went  into  the  course  design, 
we  felt  that  SMART  was  ready  for  a  summative  test.  However,  it  was  also 
felt  that  learning  to  optimize  course  design  was  an  important  outcome  of  the 
project.  Thus,  formative  analyses  were  also  a  valuable  source  of  data  to 
add  to  our  knowledge  base  on  course  design  for  the  distance  learning 
environment . 


Method 

Participants 

Students  in  the  SMART  class  were  recruited  from  a  pool  of  soldiers  who 
(1)  needed  to  take  Phase  III  of  EOAC  and  (2)  had  already  taken  Phase  I.  All 
interested  students  who  were  available  in  the  desired  time  frame  were 
permitted  to  take  the  course. 

Fourteen  students  began  the  course.  These  students  were  all 
Lieutenants  and  Captains  and  all  had  at  least  some  college  education  (12  of 
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the  14  were  college  graduates),  although  only  about  half  majored  in 
engineering.  Their  average  age  was  34,  and  only  one  was  female. 

The  control  group  consisted  of  RC  officers  who  took  Module  6  of  EOAC  in 
residence  at  the  Engineer  School  during  fiscal  years  1987,  1988,  and  1989. 
Data  on  test  (n=370),  homework  (n=171),  and  course  culminating  practical 
exercise  performance  (n=165)  were  collected  and  pre-  and  post-surveys  were 
administered  to  a  subset  of  the  RC  students  (n=49). 

Material s 


Course  Description 

The  course  consisted  of  Module  6  of  the  Engineer  Officer  Advanced 
Course  (EOAC)  as  it  was  taught  by  the  USAES  (U.S.  Army  Engineer  School)  in 
1987-88.  EOAC  is  a  mid-level  course  (i.e.,  between  senior  service  school 
and  technical  MOS  training)  for  officers  in  the  grades  of  lieutenant  and 
captain.  Engineer  officers  must  complete  EOAC  before  they  can  be  promoted 
to  major.  It  was  felt  that  the  participants  in  this  group  would  be  mature, 
be  committed  to  the  Reserve  Components,  and  be  more  apt  to  have  either 
personal  computer  experience  or  the  ability  to  learn  quickly  to  use 
computers.  Additionally,  since  68%  of  all  Army  engineers  are  in  the  RC, 
there  was  a  clear  potential  benefit  of  enhancing  RC  training  within  the 
Branch . 

Further,  the  Program  of  Instruction  (POI)  for  EOAC  includes  a  mix  of 
technical  and  leadership  objectives.  There  is  a  stated  instructional  goal 
for  the  development  of  group  skills  through  small  group  instruction  and 
group  interaction,  an  ideal  application  of  ACC. 

When  taught  in  residence,  Module  6  of  EOAC  consists  of  a  mix  of 
classroom  lectures,  lead  through  practical  exercises,  computer-assisted 
instruction,  small  group  instruction,  and  a  culminating  practical  exercise. 
The  module  is  taught  over  a  two  week  period.  Both  RC  and  AC  students 
participate  in  the  class.  In  the  time  period  during  which  data  were 
collected  for  this  project,  55%  of  all  resident  students  were  RC.  However, 
the  AC/RC  mix  in  resident  programs  is  highly  variable,  and  depends  on 
training  budgets  and  slots  available. 

Nine  topics  were  taught  in  Module  6: 

--  Rear  Operations 

--  Airfield  Damage  Repair 

--  Military  Petroleum  Pipelines 

--  Allied  Presentation  (Australia) 

--  Asphalt  Production 
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--  Army  Writing  Program 

--  Flexible  Pavements 

--  Roads  and  Airfields  and 

--  Bridging. 

At  the  end  of  the  module,  a  course  culminating  practical  exercise  is 
conducted.  This  exercise,  which  is  referred  to  as  a  "capstone"  by  the 
Engineer  School,  is  a  group  activity  which  requires  students  to  integrate 
technical  information  learned  throughout  the  module  and  to  coordinate 
efforts  to  share  limited  resources.  Thus,  students  are  presented  with  a 
series  of  construction  directives  which  require  technical  solutions  to 
bridging,  roads  and  airfields,  and  similar  problems.  Additionally,  students 
are  given  a  time  frame  over  which  their  designs  must  be  able  to  be 
implemented,  and  a  list  of  personnel  and  equipment  available  to  implement 
them.  Group  members  must  negotiate  with  one  another  for  these  resources  so 
that  both  individual  directives  and  the  entire  project  can  be  completed 
within  the  stated  schedule.  The  culmination  of  the  exercise  is  a  briefing 
to  a  field  grade  officer,  in  which  design  alternatives,  scheduling 
constraints,  and  recommendations  are  presented. 

Course  Design 

The  Resident  School  structure  and  EOAC  contact  were  adopted,  as  they 
were  judged  to  be  the  best  training  the  Army  had  to  offer.  Thus,  materials 
from  the  resident  course  were  adapted  for  use  in  the  SMART  environment. 
Procedures  for  accomplishing  such  an  adaptation  are  fully  discussed  in 
"Recommendations  for  Course  Development  and  Implementation  of  SMART  Courses" 
(Hahn,  Harbour,  Schurman,  Daveline,  and  Wells,  1990.) 

Course  design  was  guided  by  the  literature  and  the  results  of  our  pilot 
project.  Details  of  the  course  design  parameters  are  given  below. 

Time  requirements  and  course  duration.  Students  were  permitted  to 
participate  in  the  course  only  if  they  would  agree  to  spend  a  minimum  of 
eight  hours  per  week.  Total  course  duration  was  set  to  accommodate  an  eight 
hour  per  week  schedule. 

Class  size.  Class  size  was  dictated  by  student  availability.  However, 
different  types  of  activities  were  structured  to  involve  different  numbers 
of  students.  For  example,  all  students  participated  in  classroom 
discussions,  but  students  were  divided  into  groups  of  four  to  seven  to  work 
on  group  activities. 

Course  structure.  The  course  was  designed  to  minimize  student  anxiety 
by  beginning  with  easy,  short  lessons.  Generally,  lesson  activities  were 
kept  as  short  as  possible.  Often,  several  activities  were  used  to  cover  the 
material  presented  in  a  single  lecture  at  the  resident  course. 
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Variety  of  media.  Several  different  media  (including  paper-based 
materials,  CAIs,  storyboards,  video-tapes,  spreadsheet  exercises,  and 
synchronous  and  asynchronous  discussions)  were  used  in  the  course.  The  main 
constraint  imposed  in  media  selection  was  that  all  media  had  to  be  suitable 
for  in-home  presentation. 

Learning  skills  and  strategies.  A  listing  of  the  tasks  and  the 
sequence  in  which  they  should  be  completed  were  used  to  help  assure  that 
students  used  materials  in  their  logical  order.  Further,  structured  tasks, 
notetaking  aids,  "mandatory"  time  on  task,  and  frequent  checks  on 
understanding  were  used  to  increase  cognitive  processing. 

Feedback.  Many  of  the  learning  activities  were  designed  to  provide 
immediate  feedback  during  or  upon  completion  of  the  activity.  For  example, 
performance  feedback  was  provided  throughout  CAI  lessons.  Paper-based 
activities  and  storyboards  were  usually  followed  by  a  computer-scored  quiz. 
Some  activities  required  instructor  grading;  instructors  were  required  to 
provide  24  hour  turnaround  on  grading  these  activities.  Further, 
instructors  were  expected  to  address  all  student  questions  or  comments 
within  24  hours. 

Motivational  strategies.  A  variety  of  motivational  techniques  were 
used  in  an  attempt  to  enhance  motivation  during  the  course.  These  included: 

(1)  Providing  feedback  to  students  regarding  their  progress  in  the 
course.  The  instructor  used  a  set  of  decision  rules  for 
determining  how  close  to  the  schedule  students  were  keeping. 
Students  were  given  messages  ranging  in  tone  from  "You're  right  on 
schedule.  Keep  up  the  good  work"  to  "You're  more  than  25% 
behind.  We  need  to  write  a  'contract'  for  you  to  catch  up.  If 
you  don't  catch  up,  you'll  be  dropped  from  the  course." 

(2)  Offering  bonus  points  for  on-time  completion  of  the  course. 

(3)  Reducing  the  maximum  point  value  for  late  assignments  to  75  rather 
than  100  points. 

(4)  Posting  an  honor  roll  of  each  week's  "best  achievers." 

(5)  Providing  feedback  on  participation  in  team  projects.  Students 
were  shown  how  their  participation  in  group  exercises  compared  to 
that  of  other  team  members.  Further,  each  team  was  shown  how 
their  team's  performance  compared  to  that  of  the  other  team. 

(6)  Implementing  a  buddy  system  in  which  two  students  were  assigned  as 
partners  to  assist  one  another  throughout  the  course. 
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(7)  Promising  to  send  letters  of  commendation  and/or  appreciation  from 
the  Assistant  Commandant  of  the  USAES  to  students  after  course 
completion.  The  type  of  letter  sent  was  determined  by  the 
student's  course  participation. 

Instructor  characteristics.  If  the  Army  were  to  implement  SMART  for  RC 
training,  it  is  possible  that  part-time  members  of  the  RC  would  be  employed 
as  instructors.  Our  pilot  project  experience,  however,  was  only  with  a 
full-time  instructor.  Thus,  we  had  no  information  as  to  whether  and  how 
well  (in  terms  of  the  outcome  measures)  part-time  instructors  could  be  used 
in  the  SMART  environment. 

Hence,  in  the  implementation  test,  both  full-time  and  part-time 
instructors  were  used.  Five  Reserve  Component  officers  were  assigned  as 
instructors.  One  of  these  was  full-time  and  served  as  the  primary 
instructor,  backing  up  the  other  instructors  as  needed  to  ensure  the  quality 
of  the  instruction  provided  to  the  students.  He  had  successfully  completed 
the  EOAC  course  and  had  prior  experience  with  SMART  as  the  instructor  in  the 
pilot  project. 

The  other  four  instructors  were  assigned  part-time  in  various 
capacities  at  different  times  during  the  course.  The  roles  assigned 
included  instructor,  team  leader,  and  course  culminating  practical  exercise 
moderator.  All  instructors  received  40  hours  of  training  in  SMART 
instruction  prior  to  the  beginning  of  the  course. 

Support  communications.  A  free  hotline  was  set  up  for  the  duration  of 
the  course.  Students  experiencing  difficulties  could  call  this  hotline  for 
help  24  hours  a  day.  When  the  hotline  was  not  staffed,  messages  were 
recorded  and  help  was  provided  as  soon  as  possible.  Further,  students  had 
free  telephone  access  to  the  primary  instructor  for  the  duration  of  the 
course.  He  provided  help  on  hardware  and  software  problems,  course  content, 
and  personal  difficulties. 


Procedure 


As  is  shown  in  Figure  4,  the  course  was  implemented  in  three  phases. 
Several  variables,  including  (1)  group  interaction  opportunities  (individual 
vs  group  instruction),  (2)  pacing  (individual  self-set  schedule  vs 
externally  imposed  group  paced  deadlines),  and  (3)  type  of  course 
requirements  (content  orientation  in  both  Phases  1  and  2  vs  process 
orientation  in  Phase  3),  were  altered  across  these  phases.  Content  oriented 
course  materials  are  those  which  focus  on  the  acquisition  of  topical 
material.  Process  oriented  course  materials  are  those  which  focus  on  the 
application  of  content  knowledge,  as  in  the  course  culminating  practical 
exercise. 


21 


Phase  1  Phase  2  Phase  3 


Figure  4.  Three  phases  of  SMART 


Phase  1  had  the  following  characteristics: 

(1)  all  activities  were  designed  to  be  completed  individually,  with  no 
group  interaction  required; 

(2)  students  were  permitted  to  work  at  their  own  pace  as  long  as  they 
conformed  to  the  minimum  time  requirements  of  the  course;  and 

(3)  course  materials  were  content-oriented,  teaching  the  following 
topics:  Rear  Operations,  Airfield  Damage  Repair,  Military  Petroleum 
Pipelines,  the  Allied  Presentation,  and  Asphalt  Production. 

The  characteristics  of  Phase  2  were  as  follows: 

(1)  both  individual  and  group  activities  were  used; 

(2)  students  were  given  a  deadline  for  each  topic;  further,  group 
activities  required  that  students  be  up  to  date  with  their  course  work  to 
participate,  thus,  serving  as  a  pacing  aid; 

(3)  course  materials  were  content-oriented,  teaching  the  Army  Writing 
Program,  Flexible  Pavements,  Roads  and  Airfields,  and  Bridging. 

Phase  3,  the  course  culminating  practical  exercise  described 
previously,  consisted  of: 

(1)  a  group  activity  with 

(2)  an  externally  imposed  deadline  and 

(3)  a  process  orientation. 

The  SMART  software  was  tailored  to  the  functional  requirements  of  each 
phase.  For  example,  in  Phase  1,  students  were  allowed  to  have  social 
contact  with  one  another  in  the  break  room,  but  content-oriented 
interactions  were  limited  to  those  between  the  student  and  the  instructor. 
In  Phase  2,  content-oriented  interactions  were  unlimited.  A  full 
description  of  the  software  implementation  for  each  phase  can  be  found  in 
Appendix  B. 
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As  shown  in  Table  5,  summative  data  were  collected  during  each  phase  of 
SMART.  In  addition,  summative  data  were  collected  on  Resident  RC  students 
who  took  EOAC  during  the  time  period  that  the  SMART  course  was  being 
developed  and  implemented. 


Analysis  and  Results 
Summative  Findings 

Two  types  of  summative  comparisons,  external  and  internal,  were 
conducted.  External  analyses  compared  throughput  and  performance  data  from 
the  SMART  test  vs  data  from  the  Engineer  School  resident  version  of 
Module  6.  Internal  comparisons  looked  at  throughput,  performance,  and 
acceptability  (survey  and  interview)  data  collected  in  the  two 
content-oriented  phases  of  the  SMART  implementation  test.  Comparisons  were 
also  performed  among  the  phases  to  determine  the  impact  of  group 
interaction,  pacing  and  deadlines,  and  type  of  course  requirements.  Survey, 
interview,  and  online  data  were  used  in  those  comparisons. 

External  Comparisons 

Throughput.  Clearly,  resident  training  is  superior  to  SMART  training 
with  respect  to  both  the  duration  of  time  needed  for  training  and  the 
percentage  of  students  who  complete  the  training.  Completion  of  the 
equivalent  amount  of  content  material  to  that  taught  in  our  course  is 
accomplished  in  two  weeks  at  resident  school.  SMART  administration  took  31 
weeks.  Further,  the  drop-out  rate  at  the  resident  school  is  quite  low  (5% 
or  less). 

Both  phases  of  SMART  training  had  a  fairly  large  percentage  of  students 
complete  training.  Of  the  14  students  who  started  Phase  1,  only  one 
dropped,  for  a  drop-out  rate  of  7%.  In  Phase  2,  two  of  the  starters,  or 
15%,  dropped.  Another  two  students  stayed  in  the  course  for  the  entirety  of 
Phase  2,  but  failed  to  take  all  of  the  exams  required  for  course 
completion.  Even  counting  these  two  students,  though,  64%  of  those  who 
started  the  course  completed  it. 

Performance.  Four  types  of  performance  comparisons  were  made  between 
resident  and  SMART  students.  These  included  test  performance,  homework 
performance,  course  culminating  practical  exercise  performance,  and 
perceived  skill  ratings.  Test  performance  was  the  most  difficult  of  these 
comparisons  because  it  was  not  possible  to  equate  testing  conditions  for  the 
two  groups. 

Students  in  the  SMART  course  were  given  test  questions  pertinent  to 
each  subcourse  immediately  following  each  topic.  These  questions  were  the 
same  as  had  been  used  in  the  resident  end  of  course  test  at  the  time  we 
began  course  conversion.  However,  by  the  time  data  collection  began  for 
resident  school  students,  the  resident  course  tests  had  been  changed.  The 
new  test  covered  the  same  learning  objectives  as  the  previous  test,  but 
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Table  5 


Summative  Dependent  Measures 


Measure 

How  Collected 

How  Used 

Course  duration 

Measured  length  of  time 
needed  to  complete  course 
for  Resident  vs  SMART  and 
length  of  time  to  complete 
each  phase  in  SMART 

Resident  vs  SMART 
comparison  and 
Phase  1  vs  Phase  2 
comparison  within 
SMART 

%  Completion 

Measured  number  of 
students  completing  vs 
number  of  students 
starting 

Resident  vs  SMART 
comparison 

End  of  course  test 

SMART  students  completed 
a  subcourse  test  at  the 
end  of  each  topic; 

Resident  students  were 
tested  on  all  subcourses 
at  the  end  of  the  module 

Resident  vs  SMART 
compari son 

Homework  scores 

Both  SMART  and  Resident 
students  completed 
homework  assignments 
throughout  the  course 

Resident  vs  SMART 
compari son 

Course  culminating 
practical  exercise 
scores 

Both  SMART  and  Resident 
students  completed  a 
graded  culminating 
practical  exercise  at 
the  end  of  the  module 

Resident  vs  SMART 
compari son 

Skill  ratings 

Both  SMART  and  Resident 
students  completed  a 
pencil  and  paper  survey 
of  their  perceived  skill 
on  each  Module  6  topic 
before  and  after  taking 
the  course;  pre-post 
comparisons  indicated 
perceived  learning 

Resident  vs  SMART 
comparison  and 
comparison  of 
Phases  1  and  2 
within  SMART 

24 


Table  5  (continued) 


Measure 

How  Collected 

How  Used 

Scores  on  each  activity 

For  SMART  students,  each 
learning  activity  was 
graded 

Comparison  of 
performance  in 

Phase  1  vs  Phase  2 

Perceived  acceptability 

SMART  student  opinions 
on  course  acceptability 
were  polled  using  a 
post-course  survey, 
interviews,  and 
comments  placed  online 
by  students 

Comparison  of 
acceptabil ity  of 

Phase  1  vs  Phase  2 

Reasons  for  dropping 

Survey  administered  to 
SMART  students  who 
dropped  the  course 

As  an  indicator 
of  acceptability 

Perception  of  group 
interaction 

SMART  student  opinions 
on  group  interaction 
were  polled  using  a 
post-course  survey, 
interviews,  and 
comments  placed  online 
by  students 

Comparison  of 
group  vs  individual 
learning 

Type  and  content  of 
online  interactions 

Onl ine  dialog  was 
analyzed  to  determine 
the  amount  of  student- 
student,  student- 
instructor,  and 
instructor-student 
dialog,  and  the  content 
of  those  interactions 

Comparison  of 
interaction 
patterns  given 
group  vs 
individual 
learning  modes  in 
Phase  1  vs  Phase  2 
and  content  vs 
process  orientation 
of  Phase  2  vs  Phase  3 

Perception  of  pacing 
and  deadlines 

SMART  student  opinions 
on  pacing  and  deadlines 
were  polled  using  a 
post-course  survey, 
interviews,  and 
comments  placed  online 
by  students 

Comparison  of 
individual  self¬ 
pacing  vs  group 
external  pacing 
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Table  5  (continued) 


Measure 

How  Collected 

How  Used 

Actual  vs  expected 
completion  time 

For  each  topic,  a  record 
was  kept  of  when  each 
student  completed  vs 
when  completion  was 
expected  given  an  8  hr 
work  week  or  deadlines, 
as  appropriate 

Comparison  of 
individual  self¬ 
pacing  vs  group 
external  pacing 

Perception  of  content 
and  process  oriented 
activities 

SMART  student  opinions 
on  content  and  process 
oriented  activities  were 
polled  using  a  post¬ 
course  survey,  interviews, 
and  comments  placed 
online  by  students 

Comparison  of 
content  vs  process 
learning  activities 
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different  questions  were  used.  Hence,  SMART  subcourse  scores  and  resident 
school  end  of  course  scores  are  similar  with  respect  to  the  learning  to 
testing  interval  (i.e.,  both  are  post-tests  administered  immediately 
following  content  presentation)  but  are  dissimilar  with  respect  to  the 
specific  questions  asked.  Because  the  questions  on  both  exams  do  test  the 
same  learning  objectives,  however,  this  is  probably  the  best  comparison 
available. 

Based  on  this  comparison,  there  is  no  significant  difference  in  the 
test  performance  of  SMART  and  resident  students  (£  (1,  377)  =  1.80, 
p  <  0.18).  In  fact,  mean  scores  of  the  SMART  students  are  higher  (although 
not  significantly  so)  than  those  of  the  resident  students  (91.96  vs  86.44). 

Further,  100%  of  the  SMART  completers  achieved  passing  scores  on  their 
criterion  test  on  the  first  attempt;  only  93%  of  the  resident,  school 
students  in  our  sample  achieved  a  passing  grade  (70  or  better)  on  their 
first  try  at  the  exam. 

A  multivariate  analysis  of  variance  was  performed  on  the  homework 
scores  of  the  two  groups.  This  MANOVA  showed  a  marginally  significant 
difference  in  performance  of  the  two  groups  on  their  overall  homework  scores 
(Wilk's  Lambda  =  0.95,  F  =  (4,175)  2.302,  p  <  0.06).  However,  univariate 
analyses  of  variance  on  the  individual  homework  assignments  showed  no 
significant  differences  between  the  two  groups.  Table  6  shows  mean  homework 
scores  for  the  two  groups. 

Course  culminating  practical  exercise  performance  was  also  compared  for 
the  two  groups.  An  analysis  of  variance  was  conducted,  using  the  practical 
exercise  scores  as  the  measure  of  interest.  No  significant  differences  were 
found  on  the  grades  of  the  two  groups,  which  averaged  90.44  and  89.99  for 
SMART  and  Resident  School,  respectively. 

Both  SMART  and  resident  students  also  provided  ratings  of  their  skill 
level  in  the  content  areas  both  before  and  after  taking  the  course. 
Pre-course  skill  ratings  did  not  differ  between  the  two  groups. 

Pre-post  differences  were  analyzed  to  determine  the  perceived 
performance  benefits  of  the  two  groups.  A  MANOVA  performed  on  these 

Table  6 

Homework  Scores _ 


ToDic 

Resident 

SMART 

Mean 

Mean 

Flexible  Pavements 

91.73 

83.33 

Bridging 

91.31 

82.56 

Roads  and  Airfields 

95.78 

89.70 

Pipel ines 

95.39 

88.33 
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difference  scores  was  marginally  significant  (Wilks  Lambda  =  0.49,  £ 

(15,28)=  1.96,  £  <  0.06).  Univariate  analyses  of  variance  showed 
statistically  significant  differences  between  the  two  groups  with  respect  to 
perceived  performance  benefits  on  rear  operations  (F  (1,42)  =  5.73,  £  < 
0.0212),  airfield  damage  repair  (F  (1,42)  =  13.40,  £  <  0.0007),  pipelines  (F 
(1,42)  =  4.69,  £  <  0.036),  flexible  pavements  (F  (1,42)  =  8.67,  £  <  0.0052), 
and  roads  and  airfields  (F  (1,42)  =  15.42,  £  <  0.0003).  In  all  cases,  SMART 
students  perceived  a  greater  learning  benefit  than  resident  school 
students.  Table  7  shows  means  and  standard  deviations  of  the  pre-post 
differences. 

Table  7 


Pre-post  Differences  on  Skill  Ratings 


Content  Area 

Resident  Mean  (SD) 

SMART  Mean  f SD) 

Rear  Operations 

0.136  (1.13) 

1.125  (0.64) 

Airfield  Damage  Repair 

0.409  (1.13) 

1.875  (0.83) 

Pipel ines 

0.452  (1.13) 

1.500  (1.60) 

Asphalt  Production 

1.250  (1.08) 

1.875  (0.99) 

Flexible  Pavements 

0.932  (1.07) 

2.125  (0.99) 

Bridging 

0.500  (1.08) 

0.625  (0.92) 

Roads  and  Airfields 

0.809  (0.99) 

2.250  (1.17) 

Note:  Scores  were  obtained 

by  subtracting  pre-course 

ratings  from 

post-course  ratings. 

Ratings  were  based  on  a 

1-5  scale. 

Internal  Comparisons 

As  explained  in  the  "Procedure"  section,  the  course  was  divided  into  3 
phases  differing  in  type  of  pacing,  group  interaction,  and  content,  the 
following  analyses  highlight  differences  found  among  those  phases. 

Throughput .  With  regard  to  the  time  required  for  completion,  as 
compared  to  expected  completion  time,  Phase  2  was  clearly  superior  to 
Phase  1.  Students  took  1.2  times  longer  than  expected  to  complete  Phase  2 
and  2.4  times  longer  than  expected  to  complete  Phase  1.  As  will  be 
discussed  subsequently,  the  use  of  deadlines  is  primarily  responsible  for 
the  better  course  progress  in  Phase  2. 

Performance.  Table  8  shows  the  average  scores  on  each  graded  activity 
throughout  the  course.  Although  it  would  appear  that  students  performed 
better  on  the  activities  presented  in  Phase  1  than  those  presented  in 
Phase  2,  this  is  probably  more  a  function  of  lesson  content  than  any 
differences  in  the  phases  themselves.  Bridging,  on  which  scores  were 
lowest,  is  generally  considered  to  be  the  most  difficult  topic  addressed  in 
the  course. 
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Table  8 


Scores  on  Graded  Activities _ 

Activity  Mean  Score  Standard 

Deviation 


Phase  1 

Rear  Operations  Exam 
Airfield  Damage  Repair  Exam 
Pipelines  Homework 
Pipelines  Exam 
Asphalt  Production  Exam 

Phase  2 

Bridging  Homework 
Bridging  Exam 

Roads  and  Airfields  Homework 
Roads  and  Airfields  Exam 
Flexible  Pavements  Homework 
Flexible  Pavements  Exam 


96.67 

5.00 

97.78 

6.67 

88.33 

8.66 

94.78 

8.69 

93.33 

14.14 

82.56 

27.58 

83.56 

10.29 

95.78 

3.46 

98.56 

2.24 

83.33 

9.35 

88.33 

13.31 

Students  provided  self-ratings  of  their  perceived  learning.  Prior  to 
beginning  the  course,  students  rated  their  level  of  skill  in  each  content 
area.  After  the  course  was  completed,  these  ratings  were  repeated.  The 
difference  between  the  ratings  was  taken  as  an  indicator  of  perceived 
learning  benefit.  Difference  scores  on  Phase  1  topics  were  compared  to 
those  on  Phase  2  topics.  No  significant  difference  was  found  in  students' 
perception  of  learning  benefits  between  the  phases  (t  (54)  =  -0.2354,  j)  < 
0.82). 

Acceptabil ity.  Based  on  post-course  survey  responses,  interviews,  and 
online  comments,  SMART  students  seemed  to  prefer  Phase  2  over  Phase  1 
because  of  the  group  interaction  opportunities  it  afforded.  Students  also 
expressed  a  preference  for  SMART  training  over  that  provided  in 
correspondence  or  RF  Schools.  Some  students  even  felt  that  SMART  competed 
favorably  with  resident  school. 

Students  who  left  the  course  were  questioned  about  their  reasons  for 
dropping.  Sixty  seven  percent  cited  personal  reasons,  such  as  family 
commitments  or  the  need  to  complete  the  course  in  a  certain  time  frame,  as 
the  reason  that  they  dropped  the  course.  Students  who  were 
dissatisfied  with  the  course  itself  were  generally  concerned  over  the  time 
commitment  required,  rather  than  technical  aspects  of  the  instruction. 

Group  interaction.  A  great  deal  of  data  was  collected  regarding  the 
number  and  types  of  interactions  which  occurred  in  the  individual  activity 
mode  (Phase  1)  vs  the  group  interaction  mode  (Phase  2).  In  general, 
interactive  patterns  changed  between  phases,  reflecting  the  group  emphasis 
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in  Phase  2.  Further,  students  reported  positive  effects  of  group 
interaction.  Detailed  results  follow. 

Interaction  patterns  definitely  differed  between  the  phases.  In 
Phase  1,  communication  between  students  was  only  allowed  in  the  break  room. 
As  Figure  5  shows,  however,  even  in  this  forum  only  25%  of  the  communication 
was  between  students.  Most  communication  in  the  Phase  occurred  between  a 
student  and  the  instructor. 

In  Phase  2,  student  to  student  communication  in  the  break  room 
accounted  for  about  22%  of  the  total  traffic,  similar  to  the  proportion  seen 
in  Phase  1.  However,  this  figure  does  not  truly  represent  the  amount  of 
student  to  student  communication  that  took  place.  As  can  be  seen  from 
Figure  6,  in  the  team  rooms  nearly  60%  of  the  interaction  that  took  place 
occurred  between  students.  The  interaction  patterns  observed  during  Phase  2 
make  sense  in  light  of  the  functions  of  the  various  rooms.  The  team  rooms 
were  set  up  as  a  place  for  groups  of  students  to  accomplish  given  tasks; 
conversely,  the  break  room  was  an  open  forum  for  all  speakers  and  the  class 
room  was  oriented  toward  student/instructor  interaction,  since  it  was  the 
place  where  students  were  to  go  to  ask  questions. 
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Note  An  entry  is  any  comment  placed  online 

Figure  5.  Types  of  interaction  in  break  room  communication  during  Phase  1 
( n*  132) 
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Team  percentage  =  team  room  entries;  Other 
percentage  =  entries  in  all  other  rooms 

Figure  6.  Types  of  interaction  during  Phase  2  (n=924  and  454  for  team  and 
other,  respectively) 


The  content  of  the  classroom  interactions  differed  between  Phases  1  and 
2.  Figure  7  shows  the  content  of  these  interactions  for  the  two  phases.  In 
Phase  1,  the  majority  of  the  discussion  centered  around  solving 
content-related  technical  problems,  motivation,  and  computer  questions  and 
problems.  In  Phase  2,  the  emphasis  shifted  to  technical  discussion, 
requests  for  help  on  technical  subjects,  and  responses  to  those  requests. 

These  changes  partially  reflect  the  shift  from  individual  to  group 
emphasis  between  the  phases.  In  Phase  1,  students  solved  technical 
exercises  on  their  own  and  discussed  them  with  the  instructor  in  the 
classroom.  In  Phase  2,  technical  exercises  were  usually  conducted  as  a  team 
effort;  thus,  the  discussion  surrounding  the  solutions  to  technical  problems 
occurred  in  the  team  rooms. 

Further,  in  Phase  1  the  instructor  worked  with  individual  students  to 
foster  motivation.  In  Phase  2,  emphasis  was  placed  on  "team  spirit"  as  a 
motivator  and  motivational  messages  were  more  likely  to  be  delivered  in  the 
team  rooms  than  in  the  more  individual -oriented  classroom. 

When  students  were  asked  about  their  preferences  for  group  or 
individual  work,  43%  of  the  respondents  said  that  they  preferred  team  work, 
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Figure  7.  Content  of  classroom  interactions  for  Phases  1  (n=206)  and  2 
(n=251) 


while  29%  said  that  they  preferred  individual  work.  Clearly,  group 
activities  are  accepted  favorably. 

Further,  all  respondents  either  agreed  or  strongly  agreed  with  the 
statement  "Working  in  teams  motivated  me  to  participate  in  the  class" 
attesting  to  the  positive  impact  of  group  interaction  on  throughput.  One 
student  commented  "Team  work  positively  impacted  motivation  as  compared  to 
the  individual  section."  Another  said:  "I'm  enjoying  the  team  exercises, 
and  I'm  learning  faster  and  more  thoroughly  by  being  able  to  share  everyone 
else's  knowledge  and  expertise." 

Pacing.  Two  different  pacing  issues  were  addressed  simultaneously: 
the  effects  of  using  individual  vs  group  activities  as  pacing  aids  and  the 
effects  of  deadlines.  General  results  were:  (1)  group  activities  require 
more  calendar  time  for  completion,  thus,  slowing  the  course;  however,  they 
also  serve  as  a  pacing  aid,  because  students  must  be  caught  up  with  previous 
course  work  in  order  to  participate;  and  (2)  deadlines  are  effective  in 
pacing  students  through  the  course.  A  detailed  explanation  of  these  results 
follows. 

Individual  activities  can  be  completed  in  less  calendar  time  than 
asynchronous  group  activities.  The  longest  individual  activity  planned  had 
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an  estimated  duration  of  three  hours.  Based  on  the  eight  hour  per  week 
participation  requirement,  it  was  necessary  to  allot  three  days  for 
completion  of  this  assignment.  Host  individual  activities  were  designed  to 
be  completed  in  one  hour.  These  were  scheduled  to  be  completed  in  one 
calendar  day. 

Time  estimates  for  asynchronous  team  assignments  ranged  from  0.5  to  1.5 
hours.  For  asynchronous  activities,  however,  teams  were  prompted  to  begin 
work  at  least  one  week  before  the  deadline.  One  average,  asynchronous 
activities  took  six  days  to  complete. 

Two  synchronous  activities,  in  which  students  were  required  to  all  work 
at  the  same  time,  were  each  scheduled  to  be  completed  in  two  hours  (each 
activity  was  completed  in  one  day).  Actual  completion  time  averaged  3.5 
hours.  While  the  synchronous  activities  were  all  completed  in  one  calendar 
day,  they  did  require  substantial  lead  time  so  that  students  could  set  up  a 
meeting  time.  On  average,  students  began  organizing  synchronous  activities 
nine  days  prior  to  their  occurrence. 

In  spite  of  the  fact  that  group  activities  slow  the  pace  of  the  course 
in  calendar  time,  they  may  also  serve  as  a  pacing  aid.  Students  reported 
that  they  tried  to  be  caught  up  on  their  individual  activities  so  that  they 
had  the  knowledge  needed  to  contribute  during  group  exercises:  "Saturday 
morning  I  discovered  our  group  project  is  due  Sunday  evening  and  Ken  and  I 
are  responsible  for  Part  2.  My  conscience  says  I  am  not  holding  up  my  end 
of  the  bargain,  so  I  have  worked  the  entire  exercise.  Part  1  and  2  to 
follow.  But  first  I  had  to  work  the  first  several  parts  of  segment  seven  so 
that  I  may  know  what  in  the  world  is  going  on." 

Very  few  other  comments  were  made  by  students  concerning  pacing  effects 
as  a  result  of  using  individual  vs  group  activities.  A  few  comments  were 
made,  however,  regarding  pacing  effects  of  asynchronous  vs  synchronous 
activities.  Online  comments  showed  a  preference  for  synchronous  activities 
since  their  use  allowed  students  to  gain  quick  closure  on  an  activity  then 
move  on  to  something  else.  However,  use  of  too  many  synchronous  activities 
would  result  in  too  rigid  a  schedule. 

The  imposition  of  deadlines  proved  to  be  critical  in  keeping  students 
working  close  to  schedule.  Figure  8  shows  student  progress  through  the 
course  during  Phase  1  when  the  course  schedule  was  self-imposed.  The 
"expected  completion"  line  reflects  the  number  of  activities  that  should 
have  been  completed  had  students  been  spending  eight  hours  per  week  on  the 
course. 

Progress  was  so  poor,  that  a  formative  change  was  made  part  way  through 
Phase  1  and  deadlines  were  imposed.  Figure  8  also  shows  that  progress 
through  Phase  1  improved  once  deadlines  were  set. 

Figure  9  compares  progress  in  Phases  1  and  2.  The  Phase  1  data  reflect 
both  the  difference  between  the  expected  number  of  days  for  topic  completion 
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if  students  were  spending  the  required  eight  hours  per  week  and  the  actual 
number  of  completion  days.  Based  on  the  positive  impact  of  imposing 
deadlines  in  Phase  1  demonstrated  above,  these  data  must  be  considered  "best 
case"  in  comparing  progress  in  Phases  1  and  2. 

The  Phase  2  data  reflects  the  difference  between  the  scheduled 
completion  time  given  the  deadline  dates  vs  the  actual  completion  time. 

Note  how  closely  the  schedule  was  adhered  to  under  the  deadline  condition  in 
Phase  2.  In  the  worst  case,  Topic  7,  students  only  missed  the  deadline  by 
an  average  of  two  days. 

Students  felt  that  deadlines  helped  them  keep  pace  with  the  course  and 
their  acceptance  of  the  deadlines  was  quite  positive.  One  student  commented 
"The  best  thing  you  did  was  to  establish  deadlines.  Once  you  did  that,  I 
would  log  on  night  after  night  until  I  got  there  or  got  ahead,  and  then  I 
would  take  time  off,  instead  of  spacing  the  thing  out.  My  goal  was  to  meet 
the  requirement,  and  then  take  time  to  do  personal  stuff." 

Type  of  course  requirements.  Based  on  a  review  of  the  online  dialog, 
it  is  clear  that  the  quality  of  performance  was  better  on  the 
content-oriented  activities  than  on  the  course  culminating  practical 
exercise.  Both  teams  got  off  to  a  very  slow  start  on  the  practical  exercise 
--  they  spent  a  large  amount  of  time  talking  about  what  they  should  do, 
rather  than  actually  doing  it.  Further,  both  teams  had  periods  of  several 
days  in  which  no  one  did  anything!  Although  students  could  have  been 
working  offline,  team  members  awaiting  input  from  them  were  unaware  of  their 
progress. 

Figure  10  shows  the  distribution  of  online  entries  dealing  with 
technical  content  (things  such  as  solutions  to  technical  problems,  technical 
discussions,  and  questions)  vs  those  dealing  with  organization  during  the 
course  culminating  practical  exercise  and  during  other  team  activities  (both 
synchronous  and  asynchronous).  Data  from  the  two  teams  were  combined,  since 
their  interaction  patterns  were  quite  similar.  Note  that  during  the  content 
activities,  students  spent  about  equal  amounts  of  effort  planning  and 
performing  activities.  During  the  practical  exercise,  more  effort  was 
devoted  to  organization  and  less  to  actually  performing  the  task. 

Teams  approached  both  the  course  culminating  practical  exercise  and  the 
content-oriented  projects  similarly,  by  having  several  (usually  two)  people 
working  on  the  same  parts  of  the  problem,  verifying  each  others'  work.  This 
method  did  not  work  well  for  the  students  in  the  practical  exercise, 
however,  because  the  teams  were  small  in  relation  to  the  amount  of  work 
required.  (See  the  discussion  on  class  size  for  more  information  on  this 
issue.) 

Post-course  surveys  showed  that  57%  of  the  students  did  not  feel  that 
their  content-oriented  group  activities  prepared  them  to  know  how  to 
interact  during  the  practical  exercise.  Post-course  interviews  revealed 
that  83%  of  those  surveyed  did  not  understand  the  goals  of  the  exercise. 
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Figure  10.  Content  of  interactions  during  team  activities  (n=290)  vs  course 
culminating  practical  exercise  (n=173) 


They  expected  it  to  be  focused  on  technical  information,  as  the  previous 
group  activities  had  been,  and  did  not  expect  the  process  orientation. 

There  is  no  clear  cut  information  available  for  judging  whether  there 
is  a  difference  in  throughput,  in  terms  of  speed  of  completion,  between 
content-oriented  and  process-oriented  activities.  Since  the  course 
culminating  practical  exercise  was  a  much  larger  exercise  than  any  of  the 
content-oriented  activities,  it  took  quite  a  bit  longer  to  complete.  We 
have  no  quantitative  basis  for  comparing  content-  and  process-oriented 
activities  of  similar  size.  However,  our  feeling  based  on  our  observations 
is  that  process-oriented  activities  would  take  longer  to  complete  because 
they  require  much  more  coordination  among  students,  as  evidenced  by  the 
increased  organizational  effort  in  the  practical  exercise.  This 
coordination  is  very  time  consuming  in  a  setting  where  the  coordinator  must 
often  wait  24  hours  or  more  before  receiving  an  answer  regarding  the 
suitability  of  the  plan  being  made. 

In  interviews,  student  reactions  regarding  throughput  issues  indicated 
that  content-oriented  activities  had  a  positive  impact  on  completion  rate. 
Students  were  motivated  to  stay  active  in  the  course  as  a  result  of  their 
participation  in  team  activities. 


36 


The  course  culminating  practical  exercise,  coming  at  the  end  of  the 
module,  had  no  effect  on  completion  rate  for  the  module.  Students  were  of 
the  opinion  that  they  would  complete  this  one  last  hurdle  no  matter  how 
difficult.  However,  judging  from  the  degree  of  burnout  reported,  it  is 
unlikely  that  students  would  have  wanted  to  begin  another  SMART  phase 
immediately  following  this  one.  In  this  respect,  the  inclusion  of 
culminating  practical  exercises  at  the  end  of  some  or  all  modules  or  phases 
may  hurt  student  retention  at  the  course  level. 

Based  on  online  comments  and  interview  data,  students  clearly  found 
group  content-oriented  activities  to  be  acceptable.  Representative  comments 
include:  "You  asked  us  to  give  our  reaction  on  how  we  liked  it  [synchronous 
discussion].  If  you  don't  mind  a  delayed  reaction,  let  me  just  say  that  it 
was  great"  and  "The  most  enjoyable  part  of  the  whole  thing  [the  course]  has 
been  the  five  other  folks  on  this  team." 

Comments  on  the  course  culminating  practical  exercise  were  not  so 
positive.  Many  students  complained  that  the  workload  during  the  practical 
exercise  was  too  great  and  that  the  capstone  briefing  was  a  demoralizing 
experience  due  at  least  in  part  to  their  lack  of  knowledge  of  what  was 
expected  of  them.  Other  students  did  feel  that  the  practical  exercise  was 
or  would  have  been  a  valuable  learning  experience  if  they  had  had  some  prior 
background  in  performing  capstones  and/or  giving  briefings. 

Summary 

SMART  classes  compare  favorably  with  resident  programs  with  respect  to 
quality  and  acceptability,  and  are  moderately  successful  with  respect  to 
throughput.  Proper  implementation  of  group  interaction,  pacing,  and  course 
requirements  is  essential  to  course  success. 

Formative  Findings 

Through  observation  of  the  course,  we  were  able  to  learn  a  great  deal 
about  the  impact  of  our  choices  regarding  course  implementation  parameters. 
In  most  cases,  the  data  gathered  are  purely  informational  and  do  not  relate 
directly  to  the  outcome  measures.  Where  data  are  available  regarding 
throughput,  performance,  or  acceptability,  however,  they  are  noted  as  such. 

Student  Characteristics 


No  significant  differences  were  found  between  the  group  of  students 
which  finished  the  course  and  the  group  which  started  but  did  not  finish. 
Comparisons  were  based  on  demographic  data,  skills  self-ratings,  and 
opinions  of  training  methods  gathered  on  the  pre-course  surveys. 

Drop-out  surveys  confirm  that  it  was  not  because  of  lack  of  background 
or  ability  that  students  elected  to  leave  the  course.  Rather,  students 
mostly  reported  dropping  out  because  the  course  did  not  meet  their  schedule 
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needs  or  for  personal  reasons,  such  as  changing  jobs  or  having  other 
commitments  on  their  time. 

Although  the  differences  are  non-significant,  the  survey  data  show  a 
trend  indicating  that  non-engineers  may  have  had  more  difficulties  with  the 
course  than  did  engineers.  Among  the  finishers,  all  of  the  students  had 
engineering-related  Army  jobs;  only  40%  of  the  non-finishers  had  Army  jobs 
which  were  engineering-related.  Similarly,  the  finishers  had  a  larger 
percentage  of  engineering  civilian  jobs,  engineering  degrees,  and  computer 
expertise  than  the  non-finishers.  Again,  however,  the  differences  are  not 
significant. 

Comparisons  were  also  made  between  the  group  of  students  which  started 
the  course  and  the  group  of  students  which  received  preliminary  instruction 
about  the  course  and  completed  the  pre-course  survey,  but  did  not  actually 
start  the  content-related  learning  activities.  One  interesting  significant 
difference  was  found  between  the  starters  and  the  non-starters.  The 
non-starters  had  a  significantly  more  positive  opinion  of  RF  Schools  than 
did  the  starters  (Satterthwaite  t  (11.9)  =  4.01,  p  <  0.0017).  Further, 
although  their  overall  opinions  of  correspondence  and  resident  courses  did 
not  differ  significantly,  the  non-starters  viewed  some  aspects  of  these 
courses  more  favorably  than  did  the  starters.  Non-starters  had  a  more 
positive  opinion  of  correspondence  with  respect  to  increasing  confidence  as 
an  officer  and  getting  timely  questions  to  answers  than  did  starters.  With 
respect  to  resident  school,  non-starters  had  a  more  favorable  impression  of 
building  professional  connections  than  did  starters.  This  suggests  that  the 
non-starters  may  have  perceived  the  availability  of  other  attractive  options 
for  course  completion. 

Further,  starters  were  more  likely  than  non-starters  (t  (19)  =  -2.14,  p 
<  0.05)  to  report  that  they  understood  the  learning  contract  given  at  the 
beginning  of  the  course.  This  contract  specified  what  was  expected  of 
course  participants  in  terms  of  time  commitment,  number  of  online  signons, 
and  the  like. 

No  other  comparison  between  starters  and  non-starters  yielded 
significant  differences  between  the  groups;  however  a  marginally  significant 
(p  <  0.06)  result  indicates  that  non-starters  worked  more  hours  of  overtime 
per  week  than  did  starters.  Hence,  the  non-starters  may  not  have  had  time 
to  participate  in  the  course. 

Time  Requirements  and  Course  Duration 

As  the  course  progressed,  it  became  clear  that  students  were  not 
completing  activities  as  quickly  as  we  had  anticipated.  Initially,  we 
thought  that  students  were  not  keeping  up  with  their  eight  hour  per  week 
time  commitment.  Online  comments  and  discussions  with  soldiers,  however, 
contradicted  this  belief.  Rather,  students  reported  spending,  on  average, 

16  hours  per  week  on  the  course. 
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When  queried  as  to  how  their  time  was  spent,  students  reported  spending 
about  eight  hours  per  week  on  the  learning  activities  and  another  eight 
hours  per  week  performing  administrative  tasks  such  as  uploading  and 
downloading  information  and  organizing  their  materials.  We  had  not  counted 
this  overhead  time  in  ou.  time  estimates. 

Students  found  sixteen  hours  per  week  to  be  too  high  a  workload:  "We 
are  going  at  a  fast  pace.  It  is  too  fast  for  me  and  I  know  how  to  do  this 
stuff.  I  am  currently  on  a  15  to  18  hour  a  week  pace  to  keep  up."  They 
thought  that  a  total  commitment  of  eight  hours  a  week  would  be  reasonable: 

"I  think  the  pace  should  be  about  eight  hours  a  week  for  the  whole  thing  for 
reserves  because  there  are  too  many  other  activities  going  on,  working  and 
everything  else." 

In  addition  to  the  eight  hour  per  week  requirement,  we  also  told  the 
students  that  they  must  sign  on  to  the  computer  at  least  twice  per  week. 

The  eight  hours,  then,  had  to  be  distributed  over  at  least  two  sessions. 
Students  found  this  to  be  inadequate,  and  recommended  more  signons  of 
shorter  duration:  "You  must  log  on  each  day.  Logging  on  twice  per  week  is 
not  enough.  It  seems  too  much  water  has  flown  past"  and  "At  the  beginning, 

I  would  do  it  [download  and  upload]  once  or  twice  a  week.  But  I  couldn't 
keep  up  unless  I  did  it  three  or  four  times  a  week."  This  practice, 
however,  will  add  to  course  duration,  as  there  is  a  certain  amount  of 
administrative  time  associated  with  each  log  on. 

In  analyzing  data  concerned  with  completion  times  required  for  various 
types  of  learning  activities,  we  found  that  our  time  estimates  were  more 
accurate  for  some  activity  types  than  for  others  (i.e.,  there  seems  .0  be 
less  individual  variability  in  time  requirements  for  some  types  of 
activities) . 

As  shown  by  the  standard  deviation  data  given  in  Figure  11,  storyboards 
and  videos,  with  their  inherent  pacing  qualities,  are  more  likely  to  be 
completed  in  the  time  estimated  than  are  CAIs  and  reading  assignments,  in 
which  different  students  may  progress  at  various  speeds  through  the  same 
materials.  Thus,  the  types  of  learning  activities  selected  may  have  an 
impact  on  course  duration. 

Class  Size 


Surprisingly,  students  mentioned  very  little  about  class  or  team  size, 
except  to  note  the  demise  of  a  team  member.  Review  of  online  dialog  gives 
the  strong  impression  that  our  teams  were  a  good  size  for  the 
content-oriented  team  activities  (with  at  least  five  active  members),  but 
that  they  were  too  small  for  the  course  culminating  practical  exercise. 

One  capstone  team  had  only  four  members  participating.  This  meant 
that,  in  some  cases,  one  person  had  several  assignments  and  that  there  were 
no  "spare  bodies"  available  to  verify  the  work  of  other  team  members.  Even 
the  student  leaders,  who  really  should  have  had  mainly  a  coordinating  role, 
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M  denotes  mean  difference  between  estimated  and  actual  completion 
The  distance  from  x  to  M  is  one  standard  deviation 


Figure  11.  Means  and  standard  deviations  for  the  difference  between 
estimated  and  actual  completion  times  for  different  types  of  activities 


got  involved  in  technical  work.  At  a  minimum,  it  would  seem  that  team 
composition  for  course  culminating  practical  exercises  should  be:  one 
student  for  each  technical  problem,  one  technical  coordinator,  and  one 
student  to  check  all  work.  In  our  case,  then,  at  least  six  active  members 
were  needed. 

One  student's  comment  reflects  this  problem:  "There  was  like  four  of 
us  and  we  couldn't  get  enough  people  getting  back  on  timely  things  so  you 
just  couldn't  tell  what  was  going  on.  Everyone  said  'Yeah,  I'm  doing  my 
job'  and  you  didn't  know  if  they  were  or  not.  You  had  too  many  things  to 
put  time  into.  Whenever  we  get  into  a  group  activity  like  that,  if  one 
person  let's  you  down  it  really  hurts."  The  difficulty  in  determining  class 
or  team  size,  then,  is  not  one  of  determining  sheer  numbers,  but,  rather,  of 
ensuring  a  minimum  number  of  active  participants.  In  our  case,  between 
one-quarter  and  one-third  of  team  members  did  not  participate  fully  in  the 
course  culminating  practical  exercise. 

Variety  of  Media 

Student  comments  revealed  that  there  were  almost  as  many  opinions  of 
the  various  media  as  there  were  soldiers.  Two  of  the  media  selections, 
however,  were  universally  disliked:  reading  assignments  from  TMs  and  FMs 
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were  perceived  as  boring  and  difficult;  the  combination  of  videotape  and 
notetaking  guide  was  disliked  because  of  the  need  to  constantly  stop  and 
start  the  film. 

One  other  point  about  media  arose  from  student  comments:  CAI  packages 
must  be  selected  so  as  to  accommodate  a  wide  range  of  answers,  particularly 
if  students  are  expected  to  read  from  nomographs  and/or  interpolate.  The 
package  we  used  allowed  only  four  options  for  a  correct  answer. 

Consequently,  students  were  being  told  that  their  answer  was  incorrect  if  it 
was  not  one  of  the  four  programmed  possibilities.  Sometimes,  answers  which 
were  off  by  only  .001  were  marked  wrong.  Needless  to  say,  this  was  very 
frustrating  for  the  students:  "If  you  didn't  get  exactly  the  number  in  the 
program  the  answer  is  wrong.  For  mathematical  problems  that's  great,  but  my 
eyeballs  are  not  calibrated  exactly  like  someone  else's  and  therefore  I'm 
wrong.  When  you're  talking  about  interpolating  ten  numbers  within  a  1/8 
inch  space  I  feel  there  should  be  a  little  bit  of  room  on  either  side  of 
your  answers. 

Finally,  students  appreciated  the  use  of  some  paper  and  pencil 
exercises:  "For  some  reason,  I  feel  more  comfortable  with  paper  and  pencil 
than  putting  my  answers  directly  on  the  screen.  I  guess  that  the  computer 
has  something  of  a  finality  about  it.  But  you  can  always  erase  the 
pencil."  Student  comments  also  stressed  the  need  to  ease  into  computer 
skills  slowly,  since  the  fear  of  the  computer  must  be  overcome:  "I  was  very 
intimidated  by  the  computer  at  first.  Not  being  computer  literate,  I  had  to 
counter  my  fear  of  doing  the  wrong  thing." 

Following  each  individual  learning  activity,  students  filled  out  a 
short  questionnaire  on  which  they  were  asked  about  the  contribution  of  the 
media  selection  to  the  ease  or  difficulty  of  learning.  The  majority  of 
students  felt  that  most  media,  with  the  exception  of  paper-based  exercises, 
made  the  technical  content  easy  to  understand.  For  paper-based  exercises, 
41%  of  respondents  said  that  they  made  the  content  easy  to  understand,  26% 
said  that  they  made  the  technical  content  difficult  to  understand,  and  33% 
had  no  opinion.  For  all  other  media,  at  least  55%  of  the  respondents  agreed 
that  the  media  made  the  technical  content  easy  to  understand. 

Learning  Skills  and  Strategies 

Comments  online  and  in  interviews  indicate  that  students  do  not  always 
read  materials  as  thoroughly  as  was  intended:  "I  would  skim  read  a  lot 
because  of  the  time  crunch  so  I  know  what  manual  to  use.  I  know  vaguely 
where  things  are  but  I  didn't  really  read  it  in  as  technical  detail  as  I 
could  have  or  maybe  should  have." 

Further,  students  were  not  always  sure  what  points  should  receive  the 
most  emphasis:  "It  would  be  extremely  helpful  if  the  activity  description 
included  some  guidance  about  what  to  look  for  and  concentrate  on. 
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Finally,  students  relied  heavily  on  the  online  listing  of  SMART  tasks 
and  became  very  confused  during  the  course  culminating  practical  exercise 
when  it  was  not  present.  Although  students  were  mailed  a  written  listing  of 
tasks  which  detailed  specific  practical  exercise  activities  for  them  to 
complete,  they  still  reported  that  they  did  not  know  how  to  proceed. 

Students  repeatedly  had  to  be  referred  to  the  written  instructions  by  the 
instructor.  Thus,  changing  the  format  of  the  cognitive  organizers  proved  to 
be  problematic. 

Feedback 

We  had  set  a  requirement  that  instructors  must  provide  feedback  and 
answer  all  questions  and  comments  within  24  hours.  In  Phase  1,  it  was  only 
possible  to  track  turnaround  in  days;  55%  of  student  comments  and  questions 
were  answered  in  one  day.  This  figure  is  biased  on  the  low  side  for  two 
reasons:  the  instructor  often  had  telephone  contact  with  the  students,  and 
answered  their  questions  on  the  phone  prior  to  putting  a  response  online; 
and  the  primary  instructor  waited  for  the  part-time  instructors  to  assume 
their  responsibilities  --  when  they  did  not,  and  more  than  one  day  elapsed, 
he  responded  to  the  students. 

In  Phase  2,  turnaround  occurred  within  24  hours  in  nearly  80%  of  the 
cases.  As  shown  in  Figure  12,  most  instructor  responses  occurred  either 
within  four  hours  of  when  the  student  posted  the  comment,  reflecting  mainly 
synchronous  discussion  during  team  activities  and  office  hours  (periods 
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Figure  12.  Turnaround  time,  in  hours,  during  Phase  2  (n=243) 
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Finally,  students  relied  heavily  on  the  online  listing  of  SMART  tasks 
and  became  very  confused  during  the  course  culminating  practical  exercise 
when  it  was  not  present.  Although  students  were  mailed  a  written  listing  of 
tasks  which  detailed  specific  practical  exercise  activities  for  them  to 
complete,  they  still  reported  that  they  did  not  know  how  to  proceed. 

Students  repeatedly  had  to  be  referred  to  the  written  instructions  by  the 
instructor.  Thus,  changing  the  format  of  the  cognitive  organizers  proved  to 
be  problematic. 

Feedback 

We  had  set  a  requirement  that  instructors  must  provide  feedback  and 
answer  all  questions  and  comments  within  24  hours.  In  Phase  1,  it  was  only 
possible  to  track  turnaround  in  days;  55%  of  student  comments  and  questions 
were  answered  in  one  day.  This  figure  is  biased  on  the  low  side  for  two 
reasons:  the  instructor  often  had  telephone  contact  with  the  students,  and 
answered  their  questions  on  the  phone  prior  to  putting  a  response  online; 
and  the  primary  instructor  waited  for  the  part-time  instructors  to  assume 
their  responsibil ities  --  when  they  did  not,  and  more  than  one  day  elapsed, 
he  responded  to  the  students. 

In  Phase  2,  turnaround  occurred  within  24  hours  in  nearly  80%  of  the 
cases  As  shown  in  Figure  12,  most  instructor  responses  occurred  either 
within  four  hours  of  when  the  student  posted  the  comment,  reflecting  mainly 
synchronous  discussion  during  team  activities  and  office  hours  (periods 


0-4  hr  4-8  hr  8-l2hr  12-16  hr  16-20  hr  20-24  hr  24-36  hr  36-48  hr  48-72  hr  >72hr 

Time  Period 

Figure  12.  Turnaround  time,  in  hours,  during  Phase  2  (n=243) 
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during  which  the  instructor  could  be  expected  to  be  online),  or  in  the  eight 
to  16  hours  post-comment  time  frame,  reflecting  that  the  primary  instructor 
worked  during  the  day  while  the  students  worked  at  night. 

Student  comments  indicate  that  24  hour  turnaround  is  often  too  slow, 
especially  when  the  student  is  working  under  time  pressure  and  needs  to  wait 
for  feedback  to  go  on.  This  was  a  source  of  frustration  for  students,  as 
indicated  by  the  following  comments:  "When  I  would  hit  a  snag,  type  up  a 
message  to  an  instructor  or  try  to  call,  not  being  able  to  get  an  immediate 
response  would  shut  me  down  for  the  night"  and  "I  want  to  get  done  with  this 
tonight  and  if  I  try  to  contact  the  instructor  I  will  be  waiting  until 
tomorrow  to  get  an  answer.  This  is  the  delay  in  response  that  constantly 
occurs.  I  can't  expect  an  instructor  to  be  available  all  the  time,  but  I 
can't  wait  to  hear  from  him." 

Obviously,  delays  in  feedback  hinder  throughput  in  terms  of  rate  of 
course  completion.  We  had  set  up  our  course  such  that  the  receipt  of 
feedback  on  a  particular  activity  was  a  prerequisite  for  beginning  the  next 
activity.  This  caused  delays  for  the  students,  who  wanted  to  work  for 
several  hours  in  a  sitting,  but  were  unable  to  because  they  could  not  get 
feedback  from  the  instructor  until  the  next  day.  Students  suggested  that 
the  specification  of  prerequisites  be  very  limited  in  future  SMART  courses. 

The  quality  of  the  feedback  provided,  however,  was  quite  good, 
resembling  the  types  of  interaction  one  would  expect  to  see  in  a 
face-to-face  setting.  The  following  is  an  example  of  part  of  a 
student-instructor  dialog  which  took  place  during  the  course: 

Student:  "I  have  been  working  on  this  PLATO  for  going  on  six  hours.  I 

do  not  understand  what  is  being  explained  to  me.  Rather  than  just  getting 

further  behind  I'm  going  on.  The  calculations  on  V  and  M  etc.  is  what  I  am 
talking  about.  Do  you  have  any  ideas  on  how  I  can  better  grasp  this 
theory?" 

Instructor:  "If  you  can  be  more  precise  about  what  you  don't 
understand  it  would  be  easier  to  help  clarify  the  problem.  The  concepts  of 
moment  and  shear  involve  how  the  forces  act  on  the  bridge.  Moment  always 
applies  to  a  lever.  It  is  defined  as  the  force  times  the  length  of  the 
lever  arm.  In  order  for  a  bridge  to  support  a  load  it  has  to  be  strong 
enough  to  resist  the  bending  moment.  The  'section  modulus'  is  a  measure  of 
a  beam's  ability  to  resist  bending.  A  moment  has  to  be  calculated  for  each 
type  of  load,  for  the  entire  bridge,  and  on  a  per  stringer  basis.  These 
values  plus  the  allowable  bending  stress  for  the  type  of  stringer  allow  you 
to  calculate  the  required  section  modulus  for  the  stringer.  You  then  select 

a  stringer  with  a  section  modulus  larger  than  the  calculated  value..." 

Student  comments  also  revealed  that  media  that  offered  immediate 
feedback  were  preferred:  "I  like  the  CAIs.  The  immediate  feedback  is 
great." 
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Quizzes,  too,  should  provide  detailed  feedback.  Our  computer 
administered  quizzes  were  designed  to  display  the  student's  overall  score, 
but  did  not  indicate  which  answers  were  incorrect.  (This  was  done  so  that 
quizzes  could  be  administered  more  than  once;  after  students  completed 
contingency  activities  they  were  often  referred  back  to  the  quiz.)  Students 
who  received  poor  grades  on  quizzes  were  uncertain  as  to  whether  they 
misunderstood  the  content  or  had  data  entry  errors:  "I  am  not  sure  I  can 
believe  a  65%  on  the  quiz  and  after  finishing  the  CAI  I  am  even  more  certain 
that  I  was  right.  Unless  I  hit  the  wrong  button.  Could  you  please  let  me 
know  which  questions  I  missed  and  what  the  recorded  answers  were?" 

Motivational  Strategies 

Review  of  the  course  dialog,  surveys,  interviews,  online  comments,  and 
the  opinions  of  the  instructor  served  as  the  basis  for  assessing  the 
motivational  strategies. 

Several  of  the  motivational  strategies  proved  to  be  quite  effective. 
These  included: 

(1)  Feedback  on  course  progress:  Providing  feedback  on  course 
progress  seemed  to  have  a  positive  impact  on  students'  motivation, 
especially  in  cases  where  the  student  had  fallen  behind.  In 
several  instances,  students  entered  into  a  contract  with  the 
instructor  to  catch  up  with  the  course  on  a  specified  schedule. 
Students  generally  did  make  an  effort  to  comply  with  these 
contracts . 

(2)  Maximum  of  75  points  for  late  papers:  This  incentive  worked  well 
in  motivating  students  to  get  assignments  done  on  time.  Only  one 
team  assignment  (out  of  12)  was  late,  and  this  was  more  a  result 
of  communication  difficulties  than  lack  of  effort.  (The  team  had 
the  assignment  finished,  but  failed  to  transmit  it  to  the 
instructor  as  their  "final"  answer.) 

Post-survey  responses  showed  overwhelming  support  for  this  method: 
85.7%  of  respondents  said  that  penalizing  late  assignments  was  an 
effective  motivator. 

Thus,  it  would  seem  that  penalizing  late  papers  had  a  positive 
impact  on  throughput,  keeping  completion  rate  on  target. 

(3)  Feedback  on  participation  in  team  projects:  The  instructor  felt 
that  this  technique  was  effective  and  caused  students  to 
participate  more  than  they  would  have  otherwise.  Online  comments 
support  this  idea.  Students  found  team  participation  to  be  highly 
motivating  and  also  used  participation  feedback  statistics  to 
guide  their  own  inputs:  "I  felt  that  I  had  to  be  right  at  the  top 
in  participation  in  group  projects  or  I  was  letting  down  the 
group. " 
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On  post-surveys,  100%  of  respondents  said  that  team  membership 
motivated  them  to  participate.  Further,  57.1%  said  that  knowing 
their  participation  rate  vs  that  of  other  team  members  was 
motivating. 

In  light  of  these  findings,  it  is  likely  that  team  feedback 
enhanced  completion  rates.  Teams  with  members  who  were  not  highly 
motivated  to  participate  would  probably  not  have  been  able  to 
complete  activities  in  a  timely  fashion.  Performance  may  also 
have  been  impacted,  since  students  knew  that  other  team  members 
would  be  reviewing  their  work. 

Instructor  Characteristics 


The  analysis  of  the  appropriateness  of  using  part-time  vs  full-time 
instructors  for  SMART  courses  must  be  based  on  a  case  study  approach  to  our 
experiences  during  the  course.  There  are  so  few  quantitative  data  on  the 
part-time  instructors  that  numerically  based  comparison  of  the  part-  vs 
full-time  instructors  is  generally  unwarranted. 

During  Phase  1,  we  had  assigned  a  part-time  instructor  to  serve  as  the 
"team  leader."  This  role  involved  counseling  students  who  were  having 
difficulties  either  with  course  content  or  personal  problems.  Only  three 
online  comments  were  recorded  from  this  person  during  the  phase.  Student's 
comments  to  him  went  unanswered.  Repeated  phone  calls  and  visits  were  made 
to  the  team  leader  to  encourage  him  to  accept  his  responsibilities.  When 
these  attempts  failed,  the  formative  decision  was  made  to  protect  the 
quality  of  the  students'  experience  with  the  team  leader  (a  position  we  felt 
to  be  crucial)  and  the  full-time  instructor  assumed  the  team  leader's 
responsibilities  for  the  remainder  of  Phase  1  and  all  of  Phase  2. 

Two  other  part-time  people  were  assigned  as  instructors  during  Phases  1 
and  2.  Of  these,  one  was  unable  to  participate  due  to  computer  problems 
resulting  from  an  irregular  power  supply.  The  full-time  instructor  assumed 
his  duties. 

The  other  part-time  instructor  worked  out  fairly  well.  He  did 
competently  respond  to  students'  comments  and  questions  and  did  assume  the 
administrative  duties  of  the  instructor  (i.e.,  grading)  during  the  topics 
for  which  he  was  responsible.  Unfortunately,  students  had  become  very 
reliant  on  the  primary  instructor,  since  he  had  been  the  person  with  whom 
they  had  had  all  their  interactions  up  to  that  point.  Hence,  he  continued 
to  receive  questions  and  phone  calls  on  topics  for  which  the  part-timer  was 
assigned  as  the  instructor. 

As  a  result  of  their  experiences  with  the  unrel iabil ity  of  the  team 
leader  and  the  part-time  instructor  who  had  power  problems,  student 
acceptance  of  the  part-time  instructors  was  quite  poor.  Students  had  the 
perception  that  only  the  full-time  instructor  could  be  depended  upon:  "I 
don't  think  the  instructors  participated  enough.  The  only  one  I  really  have 
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a  feel  for  was  Keith  [the  full  time  instructor].  The  other  ones  [part  time] 
didn't  seem  to  get  into  it  too  much.  A  lot  of  times  it  was  'Here's  your 
instructor'  and  that's  it,  so  Keith  had  to  pick  up." 

The  use  of  a  part-time  instructor  as  the  course  culminating  practical 
exercise  moderator  worked  out  reasonably  well.  The  nature  of  the  practical 
exercise  briefing  almost  requires  that  the  moderator  not  be  someone  with 
whom  the  students  have  developed  comfortable  rapport.  Thus,  the  primary 
instructor  would  not  have  been  suited  for  this  position.  The  main 
disadvantage  of  using  this  particular  part-timer  in  the  capacity  of 
practical  exercise  moderator  was  that  he  had  not  kept  up  very  well  with  the 
content  of  the  course  or  the  use  of  the  computer  conferencing  software. 
Hence,  a  good  deal  of  coaching  on  our  part  on  both  technical  content  and 
computer  use  was  needed  as  he  participated  in  the  briefing.  This  introduced 
many  delays  into  the  briefing  and  made  the  experience  quite  hectic  for  the 
researchers  and  the  primary  instructor. 

A  small  amount  of  data  was  collected  regarding  the  time  of  day  during 
which  the  part-  and  full-time  instructors  worked  on  the  course.  These  data 
are  shown  in  Figure  13.  Note  that  the  part-time  instructors  worked  on  a 
schedule  similar  to  that  of  the  students  --  most  work  was  done  at  night. 

The  full-time  instructor  worked  mainly  during  the  day.  This  pattern  may 
have  implications  for  throughput,  in  terms  of  the  rate  of  course 
completion.  The  schedule  used  by  the  part-timers  works  quite  well  if  the 
instructor  and  the  student  happen  to  be  online  at  the  same  time;  the  student 
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Figure  13.  Online  participation  times  of  part-time  (n*61)  vs  full-time 
instructors  ( n=2 12) 
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can  get  a  question  answered  or  obtain  feedback  and  move  on  that  same  night. 
If  the  instructor  and  student  do  not  "meet"  online,  however,  turnaround  time 
may  be  increased  over  that  provided  by  the  full-timer  who  can  be  online  in 
the  morning,  answering  questions  from  the  previous  night,  and  posting 
answers  that  will  be  available  when  the  student  signs  online  that  evening. 
Certainly,  consistency  in  turnaround  will  be  greater  with  the  full-timer, 
leading  to  a  more  reliable  completion  rate. 

Support  Communications 

As  shown  in  Figure  14,  the  primary  instructor  had  a  great  deal  of 
telephone  communication  with  the  students.  Phone  use  was  heaviest  during 
Phase  1;  during  Phase  2,  we  encouraged  students  to  communicate  online 
students  calling  with  questions  that  could  be  answered  online  were  told  to 
post  them  on  the  computer. 

Early  hotline  calls  to  the  instructor  regarding  hardware  and  software 
problems  were  quite  heavy.  Nearly  18%  of  all  calls  were  placed  in  the  first 
two  weeks  of  the  31  week  course. 

Over  the  duration  of  the  course,  students  placed  388  calls  to  the 
instructor.  This  averages  out  to  about  30  calls  per  student.  In 
comparison,  students  initiated  426  online  communications  with  the 
instructor,  or  about  33  comments  per  student.  Obviously,  the  telephone  was 
a  very  important  link  to  the  instructor. 


Figure  14.  Calls  to  the  instructor  throughout  the  course 
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Although  the  hotline  was  equipped  with  an  answering  machine  that  would 
take  messages  in  the  evenings  and  on  weekends  (with  someone  returning  the 
call  as  soon  as  possible),  students  noted  that  the  hotline  was  not  available 
when  they  needed  it  most.  This  caused  delays  in  their  ability  to  complete 
materials- -often  they  had  to  wait  a  day  or  more  to  get  a  solution  to  a 
software  or  hardware  problem  that  was  preventing  them  from  working  on  the 
course.  Further,  since  calls  were  usually  returned  during  the  day,  students 
were  away  from  their  computers  and  could  not  work  through  the  problem 
solution  while  on  the  phone,  introducing  further  delays  if  the  solution  did 
not  work.  Students  appreciated  having  someone  to  help  them  during  their 
hours  of  availability:  "The  hotline  was  generally  available  when  you're  not 
on  the  system  because  you're  on  the  system  late  at  night  and  on  weekends. 

You  guys  have  your  regular  office  hours  so  if  I  have  a  problem  at  night  I 
have  to  wait  till  the  next  day  to  go  to  work  and  call.  Of  course,  it  was 
'What's  the  problem'  and  I'd  go  home  and  check.  I  guess  it  was  Bob  I  ended 
up  working  with  a  lot.  He  ended  up  staying  late  in  the  evening  which  was 
super. " 

It  was  also  noted  that  more  telephone  contact  would  have  been  helpful 
between  students.  In  our  course,  we  did  not  encourage  such  communication 
for  several  reasons.  First,  we  wanted  to  maximize  use  of  the  computer 
conference  both  so  that  we  could  better  observe  student  interactions  and 
because  that  was  the  medium  of  interest  in  the  research.  Second,  we  wanted 
to  minimize  telephone  costs  and  were  afraid  that  students  would  not  restrict 
their  conversations  to  those  which  were  necessary.  Hence,  while  we  did  not 
ban  telephone  use,  we  did  not  encourage  it.  Most  students  did  not  call  one 
another  until  during  the  course  culminating  practical  exercise. 

Students  were  of  the  opinion  that  telephone  contact,  especially  early 
in  the  course,  would  have  been  helpful  in  building  rapport:  "More  of  a 
cohesion  occurred  when  we  got  on  the  phone  and  discussed  what  was  expected." 
Also,  small  points  of  coordination  can  be  accomplished  more  efficiently  on 
the  phone  than  online,  due  to  the  inherent  time  delay  of  asynchronous 
communication:  "I  think  that  if  you  do  this  in  the  future  you  should  make 
it  clear  to  people  that  if  they're  going  to  coordinate  little  things  using 
the  phone  works  best." 

Students  expressed  anxiety  due  to  not  knowing  classmates  and  not  being 
able  to  get  a  feel  for  them  quickly  online:  "I  think  calling  and  talking 
voice  to  voice  first  would  help,  personal  interaction.  Then  when  you  go  to 
the  computer  you  can  do  more.  I  felt  funny  because  I  didn't  know  these 
guys."  In  interviews,  however,  students  were  unsure  of  whether  a 
face-to-face  meeting  at  the  very  beginning  of  the  course  would  be 
beneficial.  They  were  afraid  that  the  experience  would  be  too  short  and 
that  they  would  not  have  time  to  develop  personal  relationships.  Several 
students  mentioned  the  value  of  getting  to  know  one  another  online  first 
then  having  a  face-to-face  meeting  to  cement  personal  bonds  prior  to 
beginning  a  course  culminating  practical  exercise. 
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Support  communications  other  than  telephone  are  necessary,  especially 
in  cases  where  students  must  turn  in  assignments  involving  graphics  which 
are  difficult  to  draw  on  a  computer  screen.  US  mail  is  probably  the  most 
economical  route,  however,  use  of  the  mail  involves  delays  of  perhaps  as 
long  as  a  week.  This  has  a  negative  impact  on  the  rate  of  course 
completion:  "I  have  completed  the  homework  but  will  not  be  able  to  go  on  to 
the  next  topic  until  you  get  it  in  the  mail  and  grade  it  and  post  it... Can 
we  speed  it  up  a  bit?" 

Further,  use  of  the  mail  may  serve  to  shorten  the  time  available  to 
complete  an  activity,  unless  postmark  date  (rather  than  receipt  date)  is 
taken  as  proof  that  an  assignment  was  completed  on  time.  This  may  not  be 
feasible  if  the  instructor  needs  to  have  the  materials  available  for  viewing 
as  the  students  participate  in  a  related  activity,  such  as  a  course 
culminating  practical  exercise  briefing. 

We  also  tried  allowing  students  to  transmit  assignments  via  telefax  to 
allow  them  the  maximum  time  to  complete  activities  while  still  having  the 
work  available  for  instructor  review  in  a  timely  fashion.  However,  no 
students  used  this  method.  It  is  likely  that  the  students  did  not  have  fax 
machines  readily  available  to  them.  However,  as  this  technology  becomes 
increasingly  available  it  should  also  become  more  appropriate  for  use  in 
SMART. 


Discussion 

In  this  section,  an  overall  evaluation  of  SMART  in  terms  of  throughput, 
performance,  and  acceptability  will  be  provided.  This  evaluation  will  be 
taken  from  the  viewpoint  of  what  the  Army  could  expect  in  terms  of  these 
outcome  measures  if  they  were  to  implement  a  SMART  course.  In  addition, 
lessons  learned  from  our  experience  will  be  noted.  These  should  be  helpful 
in  improving  future  implementations. 

Before  beginning  the  evaluation  of  SMART,  however,  it  is  instructive  to 
review  how  an  Army  implementation  of  a  SMART  course  might  look.  For  the 
purpose  of  illustration,  we  will  show  how  the  Engineer  Officer  Advanced 
Course  for  Reserve  Components  (EOAC-RC)  would  be  implemented  in  SMART.  EOAC 
is  being  used  as  an  example  only;  the  concepts  discussed  should  generalize 
to  any  RC  course. 

What  Would  an  Army  Implementation  of  a  SMART  Course  Look  Like? 

The  following  illustrates  how  the  three  phase  Officer  Advanced  Course 
for  Reserve  Components  (OAC-RC)  could  be  implemented  for  Engineer  branch 
officers  in  SMART.  Currently,  EOAC-RC  consists  of: 

Phase  I  (the  Reserve  Component  Company  Command  Course).  A  two 
week  phase  in  residence  at  the  Engineer  School  (USAES),  it  is  an 
intensive  technical,  tactical,  and  training  management  oriented 
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course  designed  to  provide  a  newly  assigned  or  prospective  company 
commander  with  the  skills  necessary  to  train  and  lead  the  unit. 

Phase  IIA  (common  leader  training).  This  phase  may  be  taken 
through  a  Reserve  Forces  (RF)  school,  by  correspondence,  or  as  a 
four  week  (two  module)  session  in  residence  at  the  USAES. 

Phase  1 1 6  (branch  specific  training).  Usually  taken  by 
correspondence,  it  may  also  be  taken  by  attending  a  four  week  (two 
module)  session  in  residence  at  the  USAES. 

Phase  III  (RF  school  resident  training).  Individuals  will  be 
tracked  separately  during  Phase  III  based  on  unit  of  assignment. 
Those  assigned  to  combat  engineer  units  will  take  Phase  1 1 1 A  while 
those  assigned  to  combat  heavy  battalions  or  construction  support 
companies  will  attend  Phase  1 1 16 .  Individual  Ready  Reservists  may 
choose  either  track.  USAES  resident  training  can  be  substituted 
for  either  phase,  with  1 1 1 A  requiring  four  weeks  (two  modules)  and 
1 1  IB  two  weeks  (one  module). 

SMART  could  be  implemented  at  the  course  level  such  that  EOAC-RC  would 
be  administered  almost  entirely  through  SMART.  (Note  -  it  is  important  to 
recognize  that  SMART  can  incorporate  face-to-face  instruction.)  Figure  15 
shows  a  SMART  implementation  structure. 

(A)  The  Engineer  School's  Resident  Company  Command  Course  would  be 
expanded  by  as  much  as  two  days  to  include  face-to-face  computer  training  to 
familiarize  students  with  SMART  methods  and  techniques  prior  to  entering  any 
computer-mediated  phase.  The  familiarization  would  continue  "remotely"  via 
actual  computer  conferencing  after  the  students'  departure  and  before 
beginning  another  phase. 

(B)  Each  of  the  remaining  phases  would  be  delivered  using  SMART  and  be 
broken  down  into  two  three  month  blocks.  Students  could  sign  up  for  the 
phases  in  any  order,  but  would,  once  in  a  phase,  take  the  blocks  in  a 
particular  sequence.  Students  could  elect  to  take  as  much  as  one  month  off 
between  blocks,  but  would  need  to  schedule  breaks  carefully  so  as  to 
complete  the  entire  course  within  the  prescribed  limit  (perhaps  30  months 
rather  than  the  current  24).  One  recommended  exception  would  be  that  no 
time-off  be  allowed  between  Phase  I  and  the  next  phase  to  minimize  the  decay 
of  computer  skills  taught  during  Phase  I. 

(C)  Topics  within  a  block  would  be  taken  in  sequence.  Each  topic 
might  contain  both  individual  and  group  work,  with  group  activities  normally 
occurring  in  parallel  (rather  than  serially)  with  individual  activities. 
Major  group  activities  such  as  the  EOAC  "capstone"  exercises,  which  serve  as 
the  culmination  of  a  block  or  phase,  could  be  placed  at  the  end  of  the 
section  where  they  would  logically  fit. 
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Topics: 


Group  Activity  . . . 


Individual 

Activity 


Individual 

Activity 
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(D)  Elective  engineer  topics  offered  by  the  USAES  could  be  made 
available  either  at  the  time  of  enrollment  or  following  the  main  course  if 
students  require  additional/refresher  training  and  have  access  to  a 
computer. 

In  this  implementation,  a  ceiling  for  class  size  could  be  determined 
from  the  number  of  students  expected  in  the  system  at  any  given  time.  It 
may  also  be  prudent  to  set  a  lower  limit  (e.g.,  10  students)  so  that:  (1)  a 
"critical  mass"  is  available  for  performing  group  activities  and  (2)  each 
block  can  be  taught  in  a  cost  effective  manner. 

Now,  let  us  see  what  outcomes  (throughput,  performance,  acceptability 
and  cost)  the  Army  could  expect  if  it  were  to  adopt  a  similar  SMART 
implementation. 

Throughput 

SMART  classes  will  take  longer  for  students  to  complete  than  will  the 
equivalent  resident  course.  In  our  study,  a  two  week  resident  course  took 
seven  months  to  present  via  SMART.  In  an  Army  implementation,  we  would 
expect  students  to  be  able  to  cover  this  material  in  about  five  months. 
Remember  that  we  had  additional  time  added  to  our  schedule  in  Phase  1, 
because  students  were  working  at  their  own  pace.  Given  that  students  will 
have  deadlines  in  an  Army  implementation,  and  that  some  subjects  (such  as 
the  Allied  Presentation  and  perhaps  the  Army  Writing  Program)  would  be 
offered  as  electives,  five  months  would  be  a  reasonable  completion  time. 

In  our  course,  throughput,  in  terms  of  the  percentage  of  students 
finishing,  was  inferior  to  that  in  resident  programs.  In  resident  school, 
95%  or  more  of  those  who  start  the  course  finish  it.  In  our  course,  only 
64%  of  the  starters  finished.  This  figure  is  probably  the  lower  limit  of 
what  would  be  experienced  in  an  Army  implementation  of  SMART.  Problems  that 
plagued  our  course  (such  as  software  problems  that  were  frustrating  to 
novice  computer  users  and  the  inaccurate  estimate  of  the  time  commitment 
needed  to  complete  the  course)  would  not  be  present  in  a  fully  developed 
Army  course. 

The  resident  school  does  not  have  a  problem  with  non-starters  -- 
students  who  receive  slots  in  resident  school  attend  in  all  but  the  most 
extreme  circumstances.  The  SMART  course  did  have  quite  a  few  non-starters 
(33%  of  those  who  signed  up).  Again,  however,  in  an  Army  implementation  it 
is  unlikely  that  the  problem  would  be  so  severe.  First,  delays  in  software 
development  that  required  postponing  the  start  date  of  our  course  would  not 
be  expected  in  an  Army  implementation  --  as  was  stated  previously,  this 
accounted  for  a  majority  of  the  decisions  to  drop  the  course  prior  to  the 
start  date.  Also,  our  students  had  other  alternatives  for  taking  the 
course,  namely,  correspondence.  If  the  Army  were  to  implement  SMART,  these 
other  alternatives  might  not  be  available. 
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Performance 


SMART  training  provides  the  same  quality  of  training  as  that  found  in 
resident  school.  When  performance  of  SMART  students  on  the  most  appropriate 
available  data  was  compared  to  the  performance  of  resident  school  students, 
no  significant  differences  were  found. 

Unlike  the  comparison  between  performance  of  students  in  the  pilot  test 
with  ACC  students,  this  lack  of  a  significant  difference  in  test  performance 
cannot  be  attributed  to  ceiling  effects.  Both  SMART  and  Resident  School 
test  score  data  contained  at  least  a  30  point  range. 

Finally,  SMART  students  were  significantly  more  confident  of  their 
learning  in  their  skills  assessment.  SMART  students  perceived  a  greater 
learning  benefit  than  did  resident  students. 

Acceptability 

In  spite  of  the  fact  that  students  found  the  SMART  course  to  be  very 
demanding,  they  had  positive  opinions  of  it  as  a  learning  experience. 
Virtually  all  completers  preferred  SMART  to  correspondence.  And,  several  of 
the  completers  compared  SMART  favorably  with  their  resident  experiences. 

Even  those  students  who  still  had  a  preference  for  resident  courses 
praised  the  availability  offered  by  SMART  courses:  "I  like  this  system 
better  than  correspondence  courses.  It  is  fun  to  talk  with  others  and  get 
in  to  the  humor  in  uniform  (borrowed  this  from  Reader's  Digest).  I  also 
like  to  go  to  Bel  voir  or  even  Ft.  Wood  to  take  classes  with  others.  But  it 
was  hard  for  me  to  get  away  more  than  once  in  a  few  years  to  two  summer 
camps  or  not  go  to  camp  with  the  unit  so  that  I  could  go  to  class.  This  is 
a  good  compromise." 

Throughput,  performance,  and  acceptability  will  be  maximized  only  with 
careful  attention  to  how  the  course  is  implemented.  The  following  section 
documents  the  lessons  we  learned  about  course  implementation. 

Lessons  Learned 


We  learned  a  great  deal,  both  from  the  quantitative  data  collection  and 
from  observations  of  the  impact  of  how  we  set  course  parameters.  The 
purpose  of  this  section  is  to  note  these  lessons  for  incorporation  into 
future  implementations  of  SMART.  Specific  guidelines  on  the  "how  to's"  of 
implementation  are  given  in  other  documents  produced  by  this  project.  These 
documents,  and  their  contents,  are  described  subsequently. 

Group  interaction.  Based  on  findings  reported  in  the  literature  that 
peer  interaction  contributes  to  high  morale  and  positive  learning 
experiences  in  remote  training,  it  is  reasonable  to  expect  that  courses 
which  incorporate  group  learning  activities  in  addition  to  individual 
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activities  would  result  in  higher  throughput  and  acceptability  than  courses 
based  only  on  individual  learning  activities. 

Although  group  activities  have  positive  benefits  in  terms  of  pacing  and 
enhancing  student  motivation,  they  reduce  the  students'  ability  to  set  their 
own  schedule  for  task  performance,  reducing  the  SMART  strength  of 
flexibility.  Thus,  a  mix  of  individual  and  group  activities  should  be 
planned. 

Further,  all  group  activities  should  be  planned  to  occur  in  parallel 
with  individual  activities,  rather  than  serially.  In  this  way,  students 
will  be  able  to  continue  to  progress  through  the  course  even  if  they  must 
wait  for  asynchronous  communications  from  group  members. 

Pacino.  Deadlines  are  indispensable  for  keeping  students  moving 
through  a  course.  Deadlines  must  be  set  far  enough  apart  to  give  students  a 
reasonable  amount  of  flexibility  in  the  schedule  they  will  set  for 
accomplishing  the  objectives,  but  not  so  far  apart  as  to  encourage 
procrastination. 

Group  activities  also  serve  as  a  valuable  pacing  aid.  Students  will 
try  to  be  caught  up  so  as  to  participate  fully  in  these  activities  and  not 
let  their  group  members  down.  These  types  of  assignments  should  be 
interspersed  throughout  the  course,  at  intervals  which  facilitate 
flexibility  of  scheduling  for  the  individual  student. 

Types  of  course  requirements.  Both  content-  and  process-oriented 
activities  can  be  conducted  successfully  via  SMART.  However,  if 
process-oriented  activities  come  at  the  end  of  a  content-oriented  course,  as 
was  the  case  with  our  course  culminating  practical  exercise,  special 
attention  is  required  to  ensure  that  students  have  the  proper  expectations 
regarding  the  activity.  For  the  duration  of  our  course,  everything  the 
students  had  been  taught  was  "cookbook"  and  did  not  require  integration  on 
the  students'  part.  Thus,  a  "cookbook  norm"  was  established  and  the 
practical  exercise,  which  required  a  great  deal  of  integration,  was  totally 
out  of  character  with  what  the  students  had  been  doing  previously.  Thus,  a 
clear  statement  of  the  objectives  of  the  exercises,  as  well  as  a  description 
of  how  students  should  proceed  in  order  to  meet  those  objectives,  are 
needed.  Further,  providing  students  a  break  down  of  the  grading  of  each 
aspect  of  the  exercise  may  be  effective  in  helping  them  to  place  proper 
emphasis  on  the  various  exercise  activities. 

Further,  careful  consideration  must  be  given  as  to  whether  or  not 
students  have  learned  everything  needed  to  complete  the  exercise  in  earlier 
portions  of  the  course.  Specifically,  consideration  must  be  given  to 
whether  students  have  sufficient  background  in  the  process  aspects  of  the 
exercise.  For  example,  in  our  course  students  said  that  they  did  not  know 
what  was  expected  of  them  during  the  briefing  portion  of  the  exercise,  since 
no  other  briefings  had  been  given  previously  in  the  course.  Hence,  a  dry 
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run  of  the  briefing  would  have  helped  prepare  them  with  respect  to  the 
procedures  expected. 

Finally,  student  expectations  must  be  properly  set  regarding  the  "tone" 
of  the  process-oriented  activities,  if  it  is  different  from  what  they  have 
previously  experienced  in  the  course.  Our  students  had  had  interactions 
with  the  instructor  which  were  quite  nurturing;  thus,  they  were  unpleasantly 
surprised  when  the  briefing  moderator  was  gruff  and  demanding.  Students 
felt  that  this  shift  in  tone  detracted  from  the  learning  experience. 

Student  characteristics.  Our  experience  reinforced  the  point  that 
students  in  SMART  classes  are  part-time  students  who  have  full-time  jobs, 
family  pressures,  and  other  military  duties.  SMART  classes  will  not  be 
successful  unless  they  are  flexible  enough  to  accommodate  a  variety  of 
student  schedules. 

Time  requirements  and  course  duration.  Students  felt  that  a  time 
commitment  of  eight  hours  per  week  would  be  a  reasonable  requirement,  if 
that  eight  hours  included  course  content  as  well  as  administrative  tasks 
such  as  uploading  and  downloading.  Careful  estimation  of  the  time  req»ired 
for  both  technical  and  administrative  tasks  is  critical  to  course 
throughput,  both  in  terms  of  rate  of  completion  and  numbers  of  completers. 

As  has  been  previously  stated,  the  extra  time  commitment  needed  to  complete 
our  course  was  one  reason  for  attrition  cited  by  students  who  dropped  the 
course. 

Class  size.  Short  content-oriented  activities  can  be  accomplished  by 
groups  of  five  to  seven  students  in  a  timely,  learning-effective  manner. 
Longer  process-oriented  activities  require  a  larger  number  of  participants 
--  a  minimum  of  six  students  is  needed  for  a  practical  exercise  like  the  one 
we  conducted.  A  group  of  ten  to  twelve  students  would  not  have  been 
unreasonable  for  the  exercise,  as  this  would  have  allowed  two  students  to 
work  on  each  content-oriented  problem  contained  in  the  exercise. 

Further,  in  every  class  there  will  be  a  certain  number  of  students  who 
do  not  participate  in  group  activities.  In  our  class,  one  or  two  team 
members  were  "absent"  from  each  group  exercise.  Hence,  group  composition 
must  be  set  to  help  ensure  that  a  critical  mass  of  students  is  available  for 
each  activity. 

Variety  of  media.  With  the  exception  of  TM  and  FM  readings,  all  media 
used  in  the  course  had  both  detractors  and  supporters.  No  strong 
preferences  among  the  media  were  expressed.  However,  students  did  prefer 
media  or  combinations  of  media  (such  as  a  paper-based  lesson  followed  by  a 
computer  scored  quiz)  which  provided  immediate  feedback  over  those  which 
required  instructor  grading. 

Students  also  liked  group  activities,  particularly  those  that  occurred 
in  the  synchronous  mode. 
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Learning  skills  and  strategies.  The  primary  lesson  here  is  that 
students  become  very  reliant  on  the  cognitive  organizers  provided  for  them 
in  the  course.  When  we  shifted  the  type  of  organizer  (e.g.,  from 
computer-based  to  paper-based),  students  became  confused  --  they  did  not  use 
the  paper-based  organizers  provided  for  the  practical  exercise  since  they 
had  become  so  accustomed  to  having  tasks  listed  for  them  on  the  computer. 
Thus,  organizers  provided  to  the  students  must  be  consistent  throughout  the 
course  and  must  also  be  very  specific  with  respect  to  what  is  required  of 
students. 

Also,  because  the  primary  source  of  student  anxiety  revolves  around 
unfamil iarity  with  the  computer  and  lack  of  computer  skills,  courses  should 
be  designed  to  build  computer  skills  incrementally. 

Feedback.  Students  preferred  to  have  immediate  feedback  on  learning 
activities.  The  need  to  wait  24  hours  for  feedback  on  some  activities  was 
very  frustrating.  However,  this  problem  could  be  ameliorated  by  allowing 
students  to  continue  on  to  other  activities  while  awaiting  feedback.  Also, 
the  availability  of  support  communications  to  quickly  solve  problems  is 
critical . 

Gi.cn  the  asynchronous  nature  of  computer  communications,  and  the  fact 
that  RC  students  are  likely  to  only  work  on  the  course  in  the  evenings  or  on 
weekends,  it  is  probably  not  realistic  to  expect  turnaround  to  occur  in  less 
than  24  hours.  Setting  the  instructor's  work  schedule  such  that  most  online 
work  is  done  in  the  mornings  is  adaptive  in  meeting  a  24  hour  turnaround 
goal  --  replies  to  the  previous  night's  questions  and  comments  can  be 
waiting  for  students  when  they  log  on  that  evening.  However,  some 
instructor  time  must  also  be  made  available  during  the  hours  when  students 
are  most  likely  to  be  online.  Evening  and  weekend  online  "office  hours"  may 
help  meet  this  need. 

Motivational  strategies.  Of  the  motivational  techniques  we  tried,  the 
most  successful  were  having  a  penalty  for  turning  assignments  in  late  and 
providing  feedback  about  team  participation.  We  recommend  that  these 
strategies  be  employed  in  other  SMART  implementations.  Penalties  for  late 
assignments  should  be  in  conformance  with  the  policy  of  the  appropriate 
resident  school . 

Instructor  characteristics.  As  was  previously  noted,  we  had  very  mixed 
success  with  our  part-time  instructors.  However,  our  experience  does  not 
provide  enough  data  to  recommend  for  or  against  using  part-timers.  Rather, 
more  research  is  needed  into  this  issue.  If  the  Army  were  to  use 
part-timers,  they  would  have  far  more  leverage  for  ensuring  that  the 
instructors  met  their  commitment  than  was  available  to  us.  In  these  more 
constrained  circumstances,  the  use  of  part-timers  might  be  more  successful. 

Support  communications.  The  availability  of  other  means  to  communicate 
with  the  instructor  (i.e.,  telephone)  is  critical.  Telephone  communication 
provides  a  faster  way  to  solve  student  problems  than  online  communication, 


56 


and  may  assist  students  in  meeting  tight  deadlines.  Even  more  important  is 
that  fact  that  students  experiencing  computer  difficulties  may  be  unable  to 
communicate  online.  These  students  need  an  alternative  means  of  reaching 
the  instructor  for  help. 

Students  also  expressed  concern  over  not  knowing  their  classmates. 
Getting  to  know  one  another  online  takes  a  considerable  amount  of  time. 
Telephone  contact  in  the  beginning  of  the  course  can  speed  this  process. 
Opinions  on  the  value  of  face-to-face  meetings  at  the  beginning  of  the 
course  were  mixed.  Unless  such  face-to-face  meetings  are  long  enough  for 
students  to  really  get  to  know  one  another,  the  expense  is  probably  not 
warranted. 

Like  online  interactions  with  the  instructor,  support  communications 
must  be  available  during  the  times  when  students  most  need  them  (i.e., 
evenings  and  weekends).  This  is  particularly  critical  for  students  having 
computer  problems,  as  they  need  to  be  able  to  call  for  help  when  they  have 
ready  access  to  their  machine  so  they  can  be  "walked  through"  potential 
solutions. 

It  is  also  helpful  to  provide  means  of  transmitting  graphic  data  that 
cannot  be  typed  on  a  computer  screen.  US  mail  and  fax  service  can  meet  this 
need. 


As  was  mentioned  previously,  these  lessons  learned  are  provided  to 
sensitize  readers  to  issues  that  they  need  to  consider  to  implement  a  SMART 
course.  Specific  guidance  on  the  "how  to's"  of  implementation  is  provided 
in  two  other  documents  generated  by  this  project.  These  are  described 
below. 

Guidance  Documents 


Distributed  Training  for  the  Reserve  Component:  Instructor  Handbook  for 
Computer  Conferencing  (Harbour,  Daveline,  Schurman,  Richards,  Hahn,  and 
Wells,  1990).  This  document  is  a  job  aid  for  instructors,  trainers,  and 
other  key  personnel  working  in  a  SMART  environment.  It  provides  general 
information  about  the  SMART  environment,  the  role  of  the  SMART  instructor, 
characteristics  of  adult  learners,  principles  of  learning,  and  information 
on  the  identification  of  high  risk  students.  In  addition,  "how  to" 
information  on  establishing  class  norms,  group  learning  techniques, 
individual  instructional  techniques,  student  motivation,  online  etiquette, 
and  course  preparation  is  given.  Finally,  troubleshooting  guidance  on 
hardware  and  software  problems,  the  SMART  environment  itself,  group  and 
individual  instruction,  normal  class  operations,  and  common  student  problems 
are  discussed. 

Distributed  Training  for  the  Reserve  Component:  Course  Conversion  and 
Implementation  Guidelines  (Hahn,  Harbour,  Schurman,  Daveline,  and  Wells, 
1990).  This  document  is  intended  to  provide  principles  and  context  for 
course  developers  and  managers  for  developing  and  managing  SMART  courses  in 
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accordance  with  the  SAT  (Systems  Approach  to  Training)  process.  It  provides 
recommendations  for  the  analysis,  design,  and  development  of  SMART  courses 
and  notes  additions  to  or  special  considerations  in  the  SAT  process  for 
SMART  course  development.  In  addition,  recommendations  for  implementation 
of  SMART  courses  are  given  with  respect  to:  (1)  when  to  use  SMART  courses; 

(2)  who  should  use  SMART  courses  --  instructor  and  student  requirements;  and 

(3)  how  SMART  courses  should  be  implemented  with  respect  to  technological, 
pedagogical,  and  administrative  considerations. 

Cost  Analysis  for  SMART  Implementation 

In  previous  sections  of  this  report,  it  was  demonstrated  that  SMART  is 
an  effective  training  method  with  respect  to  throughput,  performance,  and 
acceptability.  However,  effectiveness  alone  will  not  form  the  basis  for  a 
decision  as  to  whether  or  not  to  implement  a  SMART  program.  Cost,  too,  must 
be  considered.  This  section  reports  on  two  cost  analyses:  (1)  what  costs 
would  be  expected  if  the  Army  were  to  implement  SMART;  and  (2)  the 
cost-effectiveness  of  SMART  compared  to  resident  training. 

Implementation  Costs 

In  determining  costs  for  an  Army  implementation  effort,  costs  were 
extrapolated  from  our  actual  experience  and  were  based  on  the  assumption 
that  the  phases  of  the  implementation  model  described  in  the  "Discussion" 
section  are  equivalent  in  terms  of  material  presented  and  duration  to  Module 
6,  EOAC,  as  delivered  via  SMART.  Further,  it  was  assumed  that  each  phase 
outlined  in  the  model  would  have  a  full-time  instructor  plus  other 
instructional,  administrative  and  logistical  support  and  that  50  students 
would  be  taking  each  phase  at  any  given  time. 

Cost  analyses  were  divided  into  sections  for  course  conversion  and 
course  execution.  Costs  associated  with  the  conversion  of  resident 
materials  for  SMART  courses  were  derived  from  a  report  prepared  by  the 
TRADOC  Management  Engineer  Activity  titled  "Estimated  Time  Values  [ETVs]  for 
Training  Developments"  (1984).  These  costs  are  expressed  in  professional 
staff  hours.  Course  execution  costs  were  based  generally  on  the  cost 
categories  defined  by  Knapp  and  Orlansky  (1983).  These  costs  are  expressed 
in  dollars.  The  "course"  is  assumed  to  be  equivalent  to  one  module  of  EOAC 
(two  weeks  of  instruction  totalling  66  classroom  hours).  It  is  also  assumed 
that  a  software  package  would  be  available  to  implement  SMART  so  software 
development  costs  are  not  included  in  any  of  our  projections. 

Table  9  lists  the  estimate  of  the  number  of  hours  it  would  take  a 
branch  school  to  duplicate  the  course  conversion  accomplished  by  the  SMART 
project  staff.  The  conversion  represents  a  revision  of  existing  materials 
rather  than  full  scale  development.  For  purposes  of  comparison,  we  have 
also  used  the  TRADOC  document  on  ETVs  to  estimate  the  number  of  hours 
required  to  develop  the  same  program  of  instruction  from  scratch  for 
presentation  in  residence  using  traditional  instruction  methods.  All  the 
individual  categories  and  subcategories  for  full  scale  development  are  not 
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Table  9 


Estimated  Amount  of  Time  for  SMART  Conversion 

Cateaorv 

Time  (Hours! 

%  of  Conversion  Effort 

Course  Reauirements  Analysis 

10% 

Conduct  Initial  Review 

111 

Assemble  Draft  Training  Program 
Package 

180 

Coordinate  Draft  Training 

Program  Approval 

42 

Provide  Draft  Training  Program 
Implementation  Assistance 

102 

Course  Desiqn 

4% 

Select  Instructional 

Setting 

46 

Identify  Learning  Objective 
Relationship 

20 

Group  Learning  Objectives 

15 

Prepare  Practical  Exercise 

Tests 

82 

Course  Development 

61% 

Develop  Draft  Course  Management 
Plan 

101 

Revise  Draft  Course  Management 
Plan 

19 

Develop  New  Lesson  Plan 

459 

Develop  New  Student  Aids 

576 

Develop  New  Instructor  Guide 

81 

Validation  -  Presents  Lesson 

475 

Validation  -  Revises  Lesson 

325 

Proofreads  Final  Product 

553 

CBI/S1 ide  Conversion 

812 

19% 

Video  Tape  Production  * 

251 

6% 

Total 

4250 

Resident  Course  Development  Total 

11761 

*  Includes  the  hours  to  prepare 

and  supervise  the 

contract  for  a  CBI  project 

plus  the  hours  spent  by  the  ACC 

Project  contractor 

to  actually  perform  the  1 

level  CBI  conversion  required  by 

the  ACC  Project. 
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shown,  but  have  been  combined  to  produce  the  single  figure  shown  at  the 
bottom  of  the  table. 

Projected  course  execution  costs  are  shown  in  fable  10  with  separate 
columns  for  one  time  and  recurring  costs.  Each  column  was  further 
subdivided  into  contractor  and  in-house  sections.  Actual  hours  the 
contractor  spent  working  on  the  project  were  adjusted  by  cost  category  to 
eliminate  research  peculiar  activities  and  to  expand  the  projection  to  cover 
two  modules  and  the  increased  student  load  implementation  would  incur.  The 
hours  were  then  multiplied  by  the  contractor's  average  "loaded"  hourly  rate 
in  dollars  and  the  rate  calculated  for  an  in-house  effort  from  standard  pay 
tables  and  data  on  operating  costs  provided  by  TRADOC  (ATRM-159  Report). 

The  production,  equipment,  and  shipping  categories  are  the  same  for 
contractor  and  in-house  because  we  have  assumed  that  the  branch  school  would 
absorb  essentially  the  same  costs  experienced  by  the  contractor. 

The  execution  cost  categories  have  been  defined  as  follows:  (1) 
Production  --  reproduction  of  materials  and  preparations  for  shipping;  (2) 
Equipment  --  computer  hardware  and  software;  (3)  Training  --  training 
instructors,  both  face-to-face  and  online;  (4)  Shipping  --  transporting 
computers  and  materials  to  students;  and  (5)  Operating  and  Support  -- 
instructor  salaries,  computer  maintenance  and  communications,  support 
communications,  course  management,  and  test  and  evaluation. 

Cost  Effectiveness 


Table  11  shows  course  execution  costs  in  dollars  per  student  and 
dollars  per  student  per  hour  of  instruction.  Again,  an  in-house  / 
contractor  range  is  provided  for  the  SMART  implementation.  SMART  costs  were 
based  on  a  class  size  of  50  and  one  time  costs  were  amortized  over  five 
classes  because  it  can  be  expected  that  course  materials  and  equipment  would 
be  reused.  The  resident  costs  shown  are  based  on  the  ATRM-159  Report. 

Table  10 

TRADOC  SMART  Execution  Costs _ 


Execution  Cost  One  Time  $K  **  Recurring  SK 

(Contractor  /  In-House)  (Contractor  /  In-House) 


Production  16.8  /  16.8 

Equipment  49.3  /  49.3  . 

Training  46.7  /  23.8  0.6  /  0.3 

Shipping  8.0  /  8.0 

Operating  and  Support  189.0  /  96.1 

Totals  96.0  /  73.1  214.4  /  121.2 


**  It  has  been  assumed  that  new  training  and  equipment  will  be  required 
after  five  course  iterations. 
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Table  11 


Relative  Costs  of  RC  Training  Options 


Execution 

S/Student 

$/Student/Hour 

SMART 

4,672  (contracted)  /  2,716  (in-house) 

71  /  41 

Resident 

5,793 

88 

Relative  costs  alone,  however,  are  not  adequate  to  determine  the 
viability  of  SMART  for  RC  training.  Both  cost  and  effectiveness  measures 
(throughput  and  quality)  were  directly  compared. 

As  was  shown  previously,  SMART  courses  are  comparable  to  resident 
courses  with  respect  to  quality  (performance  and  acceptability).  Hence, 
since  execution  is  less  expensive  for  SMART  than  for  resident  school,  it 
would  compare  favorably  on  cost-effectiveness  (CE)  ratios  based  on  these 
measures. 

The  critical  CE  ratio,  however,  is  that  concerned  with  throughput, 
since  it  is  in  this  area  that  resident  school  surpasses  SMART.  Previously, 
we  gave  conservative  estimates  of  SMART  throughput  at  64%  and  of  resident 
throughput  at  95%.  Given  a  starting  class  size  of  50,  the  CE  ratios  can  be 
developed  using  the  following  formula: 

CE  =  (Class  Size  Percent  Throughput)/Cost  x  100% 

This  ratio  reflects  the  execution  cost  per  student  for  students  who  actually 
become  trained. 

For  SMART,  CE  (contracted)  =  0.68  and  CE  (in-house)  =  1.18.  The 
resident  CE  =  0.82  (see  Table  12).  The  SMART  CE  ranges  demonstrates  that 
SMART  is  at  least  comparable  to  residence  in  cost  effectiveness  at  even  64% 
throughput.  However,  as  has  been  previously  stated,  we  feel  that  SMART  has 
the  potential  to  achieve  better  than  64%  throughput.  If  this  is  the  case, 
SMART  will  clearly  exceed  resident  CE.  Figure  16  shows  that  even  contractor 
developed  and  conducted  SMART  training  at  64%  throughput  becomes  less  costly 
than  resident  training  after  four  course  iterations.  In  this  illustration, 
one  time  front-end  conversion/development  costs  have  been  added  to  one  time 
and  recurring  execution  costs  at  the  first  iterations  for  SMART  training. 

One  time  execution  costs  are  repeated  after  the  fifth  iteration  because  it 
has  been  assumed  that  there  will  be  a  need  both  to  train  new  instructors  due 
to  personnel  turnover  and  to  replace  worn-out  equipment.  Dollar  amounts  for 
conversion/development  were  determined  by  multiplying  the  hourly  rates 
previously  discussed  by  ETV  hours.  An  additional  amount  ($34. IK),  equal  for 
both  "contractor"  and  "in-house"  efforts,  was  added  to  cover  the  cost  of 
computer  based  instruction  (CBI)  development.  In  Table  9,  it  was  assumed 
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Table  12 

Cost-Effectiveness  Ratios 

Throughput:  SMART  64% 

Resident  95% 


50  x  .64 

Cost  Effectiveness  Ratios:  CE  (SMART  in-house)  «  .  x  100  =  1.18 

2716 

50  x  .64 

CE  (SMART  contracted)  . . x  100  =  .68 

4672 

50  x  .95 

CE  (Resident)  . . x  100  =  .82 

5793 


that  CBI  development  would  be  contracted  out  rather  than  conducted 
in-house. 


Conclusions  and  Recommendations 

In  this  report,  it  has  been  demonstrated  that  SMART  courses  are 
effective,  as  compared  to  resident  programs,  with  respect  to  performance  and 
acceptability.  While  SMART  courses  have  not  yet  achieved  the  throughput 
seen  in  resident  programs,  their  throughput  does  exceed  that  of 
correspondence  courses,  while  providing  the  same  flexibility  afforded  by 
correspondence.  Indeed,  SMART  courses  suit  students'  needs  for  flexibility 
of  scheduling  and  are  generally  more  available  to  a  wider  range  of  RC 
students  than  are  resident  programs. 

Further,  the  cost  effectiveness  of  contractor  developed  SMART  programs 
is  similar  to  that  of  resident  school,  in  spite  of  the  lower  throughput. 

SMART  is  very  effective  in  meeting  the  needs  of  the  RC  because  it  can 

overcome  geographical  dispersion  and  low  density  MOS  training 
requirements  by  bringing  students  together  remotely; 

stretch  training  resources  (instructors)  by  making  an  instructor 
in  one  location  available  to  students  in  many  locations; 

accommodate  civilian  and  personal  commitments,  due  to  its 
asynchronous  nature;  and 

provide  a  means  for  maintaining  Army  contacts  online. 

However,  SMART  courses  will  not  be  successful  unless  certain 
implementation  requirements  are  met.  These  include: 

providing  each  student  with  a  suitably  equipped  computer, 
preferably  a  portable  rather  than  a  desk  top  model; 

providing  well -supported,  well -documented  software  to  support 
communications,  conferencing,  and  special  applications; 

giving  special  attention  to  logistical  coordination,  including 
access  to  telephone  lines  and  adequate  power  supplies; 

setting  deadlines  for  completion  of  activities,  and  providing  both 
incentives  and  penalties  to  encourage  compliance; 

being  aggressive  in  the  application  of  motivational  techniques 
such  as  performance  feedback; 

orienting  students  to  the  nature  of  activities  (content  or  process 
emphasis)  and  explicitly  stating  requirements; 

scheduling  the  course  to  accommodate  civilian  jobs,  family 
commitments,  and  other  military  duties; 

--  using  a  variety  of  media,  including  both  group  and  individual 
learning  activities,  for  the  presentation  of  instruction;  and 

providing  suoport  communications  (i.e.,  a  telephone  hotline)  that 
are  available  during  hours  when  students  are  most  likely  to  be  working  on 
the  course. 


63 


Further  details  on  implementation  requirements  can  be  found  in 
"Distributed  Training  for  the  Reserve  Component:  Course  Conversion  and 
Implementation  Guidelines"  (Hahn,  Harbour,  Schurman,  Daveline,  and  Wells, 
1990). 

Questions  still  remain  regarding  the  viability  of  a  course  level 
implementation  of  SMART.  Specifically,  more  information  is  needed  on: 

Staffing  requirements:  what  mix  of  full-  and  part-time 
instructional  and  administrative  support  is  needed  to  conduct  a  course  level 
implementation  of  SMART? 

Software  interface:  what  are  the  optimal  software  requirements 
with  respect  to  communication,  delivery  of  instruction,  and  course 
administration?  and 

Class  size:  how  many  students  can  one  instructor  handle  in  a 
classroom  setting  and  in  a  team  room  setting? 

Based  on  the  results  found  in  the  implementation  test,  SMART  appears  to 
be  able  to  provide  acceptable  throughput,  performance,  and  availability  at  a 
reasonable  cost.  The  use  of  SMART  for  Army  training  should  be  further 
pursued. 
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APPENDIX  A 


A  SAMPLE  SMART  SESSION 

SMART  presents  a  very  different  milieu  than  the  face-to-face 
classroom.  The  following  descriptions  are  based  on  features  found  in  the 
software  implementation  created  for  the  EOAC  course.  They  are  intended  only 
as  an  example  of  what  SMART  software  features  might  look  like.  To 
illustrate  this,  Figure  A1  shows  what  the  SMART  student  sees  and  does  during 
the  computer  session  that  takes  place  at  2030  hours  on  Thursday. 

(A)  The  first  thing  the  student  sees  after  turning  on  the  computer  is 
an  analog  of  an  electronic  school.  This  school  contains  "rooms"  or  places 
where  various  activities  occur  --  feedback  on  tests  is  given  in  the  office; 
small  group  activities  take  place  in  the  team  room;  class  discussions  take 
place  in  the  classroom;  general  chit-chat  takes  place  in  the  break  room; 
self-contained  lessons,  such  as  CAIs  and  storyboards,  are  taken  in  the 
learning  center;  and  homework  assignments  and  tests  are  done  in  the  writing 
center.  The  school  metaphor  serves  as  an  aid  to  understanding  the  learning 
environment  for  the  student  who  is  moving  from  the  known  world  of  the 
face-to-face  classroom  to  the  unknown  world  of  SMART. 

(B)  The  task  list  serves  as  an  organizer,  whereby  the  student  selects 
activities  to  perform.  In  addition  to  selecting  self-contained  course 
requirements,  such  as  performing  a  CAI,  the  student  may  elect  to  send  or 
receive  information  to/from  other  students  and/or  the  instructor.  This  is 
done  by  "uploading"  or  "downloading"  to/from  the  host  computer.  The  task 
list  also  serves  a  course  management  function  as  it  keeps  track  of  what 
lesson  activities  the  student  still  needs  to  complete. 

(C)  A  typical  student  would  probably  first  elect  to  download 
information  from  the  host  computer.  By  downloading,  answers  to  any 
questions  that  were  posed  to  the  instructor,  messages  from  classmates,  and 
feedback  on  any  graded  activities  completed  would  be  received.  Only  that 
information  sent  to  the  host  computer  since  the  student  last  downloaded  is 
provided.  Information  is  organized  hierarchically  by  net,  item,  response, 
and  date.  Before  continuing  through  our  overview  of  the  SMART  session,  an 
explanation  of  these  concepts  is  in  order. 

A  net  is  the  computer  equivalent  of  a  room  in  the  electronic  school 
where  activities  take  place.  Information  from  similar  types  of  activities 
are  stored  together  in  a  net.  For  example,  in  Figure  A1  (C),  the  net  is  the 
CLASSROOM.  This  is  where  discussions  about  technical  topics  take  place.  As 
can  be  seen  from  the  figure,  students  are  asking  questions  about  course 
content. 

An  item  is  a  statement  or  discussion  about  a  single  subject  or 
thought.  In  Figure  A1  (C),  you  will  note  designators  such  as  14:4  and 
17:5.  The  number  to  the  left  of  the  colon  is  an  item  number  which  keys  the 
student  to  the  item  being  discussed.  In  the  example,  Item  14  is  a  technical 


A-1 


Break  Room 


Classroom 


Team  Room 


Writing 

Center 

Team 

Room 

Task  List 

Learning 

Center 

Team 

Room 

Office 

TASK 

LIST 

STATUS 

1. 

Do  reading  on  airfields 

DONE 

2. 

Do  CAI  oti  airfields 

NEED 

3. 

Review  roads  and  airfields  lessons 

NEED 

4. 

Take  roads  and  airfields  exam 

NEED 

A. 

Upload  to  hose 

B. 

Download  from  host 

Select 

menu  option 

Downloading  from  CLASSROOM: 

Feb21/89  23:08 

14:4)  John  Jones:  Help  on  activity  8 
When  I  fill  out  the  table  in  step  (b-2),  I  get  a  No 
Go  for  the  No.  10  sieve  and  Gos  for  the  rest.  The 
tables  values  are  80,  15,  25,  and  5.  The  sample 
values  are  80,  10,  15,  and  4.  Where  am  I  going 
wrong?  I'll  check  back  later  tonight  for  some 
advice  and  will  drive  on  in  the  meantime. 

Feb. 21/89  23:45 

17:5)  Mike  Smith:  I  need  help  too 
Help!  I  do  not  have  a  clue  as  to  what  is  going  on 
in  the  thickness  design  of  each  layer  in  the 
building  of  a  road  or  airfield.  It  is  not  clicking 
as  to  where  the  thickness  is  coming  from  or  how  to 
break  down  each  layer  and  know  how  thick  it  should 
be.  I  understand  the  first  part  of  the  problems, 
but  not  the  last  2  or  3  steps.  Can  anyone  help??? 

Feb22/89  00:08 

17:6)  Joan  Black:  Here  is  a  little  help  John 
John,  I  can't  help  you  a  lot  but  if  you  use  half  of 
the  months  at  31  days  and  half  at  30  you  will  come 
up  with  the  correct  answer. 
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Figure  Al.  Sample  SMART  session 


discussion  on  Road  Design,  while  Item  17  is  a  technical  discussion  on 
Airfield  Design. 

In  the  designators,  the  number  to  the  right  of  the  colon  is  a  response 
number.  A  response  is  a  comment  about  an  item  or  another  response. 

Responses  are  displayed  in  the  order  in  which  they  are  received,  as 
indicated  by  the  date  and  time  stamps  shown  in  Figure  A1  (C). 

The  excerpts  shown  in  Figure  A1  (C)  are  transcripts  from  a  real  class, 
with  only  the  names  changed.  They  represent  a  portion  of  the 
student-to-student  dialog  taking  place  in  about  a  one-hour  period  on  a  given 
evening.  Several  features  of  the  transcript  are  noteworthy: 

The  time  stamps  --  as  was  stated  previously,  SMART  students  do 
most  of  their  course  work  during  evening  and  weekend  hours;  the 
time  stamps  reflect  Eastern  time;  "John  Jones,"  a  student  who 
lives  in  Maryland,  was  working  on  the  course  at  11  p.m.; 

Organization  --  because  students  are  working  asynchronously,  even 
though  responses  17:5  and  17:6  follow  one  another  chronologically 
they  do  not  follow  logically;  response  17:6  obviously  is  a 
response  to  a  comment  posted  at  some  time  previous  to  17:5; 
further,  not  all  students  are  working  on  the  same  course  topics, 
as  evidenced  by  activity  in  two  items  which  do  not  refer  to  the 
same  content  areas;  the  result  of  this  is  that  students  must 
devote  time  and  effort  to  organizing  downloaded  information  into  a 
logical  (i.e.,  by  item)  sequence,  usually  by  printing  the 
information  and  binding  it  in  a  notebook;  and 

Peer  learning  --  these  items  are  being  used  by  the  students  to  ask 
and  answer  technical  questions;  while  the  instructor  would  have 
the  ultimate  responsibi 1 ity  of  answering  technical  questions, 
other  resources  are  available  to  the  student  with  a  question,  as 
evidenced  by  "Joan  Black's"  response;  unlike  the  face-to-face 
environment,  where  it  is  usual  for  the  instructor  to  speak  and  the 
student  to  listen,  computer  conferencing  affords  equal  air  time  to 
all  and  students  are  free  to  respond  to  one  another  in  any  public 
discussion. 

Returning  to  the  discussion  of  how  a  SMART  session  appears  to  the 
student,  let  us  examine  a  snapshot  of  a  the  student's  remaining  tasks,  as 
shown  in  Figure  A2. 

(A)  The  SMART  software  prompts  the  student  for  comments  on  items  and 
responses  that  were  just  read.  The  software  transforms  the  student's  entry 
into  language  understandable  by  the  conferencing  system,  so  the  student  does 
not  have  to  learn  the  peculiarities  of  a  particular  conferencing  system. 
Finally,  the  SMART  software  returns  the  student  to  the  task  list  when 
downloading  and  responding  are  completed. 


Figure  A2.  Continuation  of  sample  SMART  session 
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(B)  The  next  task  selected  by  the  student  is  performing  a  CAI  lesson. 
The  SMART  software  automatically  finds  and  displays  the  correct  lesson  for 
the  student.  As  the  student  moves  through  the  lesson,  information  is 
presented  and  questions  are  asked.  When  the  student  answers  a  question, 
immediate  feedback  is  given  as  to  whether  or  not  the  answer  was  correct. 

Serving  the  SMART  function  as  a  course  management  system,  the  SMART 
software  stores  the  student's  answers  and  automatically  sends  them  to  the 
instructor  when  the  student  next  selects  the  upload  function  from  the  task 
list.  In  fact,  selecting  the  upload  function  causes  all  newly  created  data, 
such  as  the  responses  shown  in  Figure  A2  (A)  and  answers  to  CAIs  and 
quizzes,  to  be  automatically  transmitted  to  the  host  computer  in  the  format 
required  by  the  conferencing  system.  Typically,  the  student  would  upload 
after  completing  all  other  computer-mediated  activities  during  a  particular 
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SMART  SOFTWARE  IMPLEMENTATION 

Because  Phases  1  and  2  differed  with  respect  to  the  availability  of 
group  interaction,  slightly  different  versions  of  the  electronic  classroom 
were  presented  via  SMART  for  the  two  phases.  Recall  from  the  previous 
discussion,  that  the  functions  of  SMART  are  to  provide  a  communication 
system,  a  combination  of  delivery  media,  and  a  course  management  system. 

The  course  management  function  was  fully  supported  in  both  phases.  In 
both  phases,  a  task  list  was  used  to  guide  student  access  to  the  learning 
activities.  This  task  list  provided  a  display  of  personal  progress  through 
the  course  for  each  student.  An  office  function  was  also  available  in  both 
phases;  this  provided  a  place  where  the  student  could  obtain  feedback  on 
grades  and  course  progress. 

To  a  lesser  extent,  a  variety  of  media  were  also  supported  in  both 
phases.  Both  phases  fully  supported  paper-based  and  locally  delivered 
electronic  media,  such  as  CAIs  and  storyboards,  through  the  learning 
center.  Phase  1  was  more  limited  than  Phase  2,  however,  in  that  no 
synchronous  or  asynchronous  group  activities  were  supported. 

This  point  ties  in  to  the  main  area  in  which  SMART  functions  were  not 
fully  supported  in  Phase  1  --  communication.  In  both  phases,  homework  and 
other  written  assignments  to  be  sent  privately  to  the  instructor  were 
performed  in  the  writing  center.  In  Phase  1,  however,  the  many-to-many 
communication  supported  by  ACC  was  curtailed  such  that  students  could  only 
hold  private  discussions  with  the  instructor  or  another  student.  No  group 
communication  was  possible. 

The  following  paragraphs  detail  the  features  of  SMART  that  were  unique 
to  each  phase. 

Phase  I 


Figure  B1  (A)  shows  the  analog  of  the  electronic  school  displayed  for 
the  students  during  Phase  1.  In  this  school,  content-related  private 
discussions  between  a  student  and  the  instructor  took,  place  in  the 
classroom.  One-on-one  conversations  on  any  topic  could  take  place  between 
students  or  between  a  student  and  the  instructor  in  the  break  room. 

Students  entered  the  school  via  the  task  list,  shown  as  Figure  B1  (B). 
Note  that  all  activities  on  the  task  list  are  individual  content-oriented 
activities.  At  the  outset  of  the  phase,  students  were  not  given  deadlines 
for  topic  completion.  Rather,  they  were  told  to  work  at  least  the  minimum 
eight  hours  per  week  and  to  complete  the  phase  as  rapidly  as  they  could. 

Phase  2 


In  Phase  2,  many-to-many  communications  were  supported  via  the  INTERACT 
feature  shown  in  Figure  B2  (A).  This  feature  allowed  for  group 
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Figure  Bl.  Phase  1  electronic  school 


communication  in  the  break  room  (on  any  topic),  class  room  (on 
content-related  subjects),  and  in  the  team  rooms  (on  group  exercises). 
Interaction  in  the  team  rooms  was  semi -private  in  that  it  was  limited  to 
team  members  and  the  instructors,  and  could  not  be  viewed  by  class  members 
who  were  not  part  of  the  team. 

Different  forms  of  private  communication  were  also  supported. 

One-on-one  discussions  could  be  held  between  students  in  the  mail  room,  a 
student  and  the  instructor  in  the  instructor  help  room,  and  a  student  and 
the  team  leader  in  the  team  leader  help  room. 

As  shown  in  Figure  B2  (B),  the  task  list  for  Phase  2  directed  students 
to  both  individual  and  group  content-oriented  activities.  At  the  outset  of 
this  phase,  students  were  given  due-dates  for  each  topic,  based  on  the  eight 
hour  per  week  time  requirement. 

Phase  3 


In  Phase  3,  all  features  present  in  Phase  2  were  available  to  the 
students.  However,  the  learning  center  was  only  used  by  students  needing  to 
review  previously  taught  content  --  no  new  instruction  was  presented  in  the 
learning  center.  Further,  the  task  list  was  not  presented  electronically. 
Rather,  to  avoid  costly  software  changes,  students  were  sent  a  paper 
syllabus  which  detailed  the  tasks  to  be  performed.  The  information  provided 
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on  the  syllabus  was  the  same  as  that  provided  electronically,  except  that 
students  had  to  manually  chart  their  progress  through  the  activities. 
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